Construction material distributing robot control method and system based on VLAS model

Abstract

The application discloses a construction material distributing robot control method and system based on a VLAS model design, which comprises the following steps: receiving a voice instruction, verifying the identity of a construction worker and generating a voice instruction embedding vector; aligning the voice instruction embedding vector with an RGB image in a shared semantic space, fusing visual and voice features, and generating a joint representation; task decomposition according to the joint representation; collecting image data of a construction site and identifying a target object, obtaining point cloud data and extracting coordinates of the target object; generating a motion trajectory of the target object according to the coordinates of the target object, motion control parameters of the material distributing robot and joint positions; converting the motion trajectory into a plurality of action instructions to drive the material distributing robot to perform actions; real-time detection of the distance between the construction worker and the mechanical arm of the material distributing robot, triggering an emergency stop when the distance exceeds a safe distance; real-time collection of the running state information of the material distributing robot, dynamic adjustment of the motion trajectory and the pouring speed. The application can control the robot to automatically distribute materials.

Description

Technical Field

[0001] This invention relates to the field of building construction, and in particular to a control method and system for a construction material placement robot based on a VLAS model. Background Technology

[0002] Currently, there are three main methods for laying materials in building construction:

[0003] Method 1 involves construction workers manually cooperating with traditional concrete placing machines to place the concrete. This method involves manually controlling the concrete placing machine to place the concrete. However, manual placement of the concrete can result in uneven distribution of the concrete, often requiring repeated placement or workers to have extensive experience to complete the placement. This method is not very efficient. After the concrete is placed by the concrete placing machine, the floor surface is prone to unevenness, requiring subsequent manual vibration or leveling to achieve a flat floor slab. This method is labor-intensive and has low construction efficiency.

[0004] Method 2 is a self-climbing concrete placing system structure in an elevator shaft. The climbing mechanism is fixedly installed in the elevator shaft of the building and can drive the hydraulic automatic placing machine to rise. The concrete placing machine can be lifted by hydraulic cylinders and automatically climbs in the elevator shaft. It relies on the control system and sensors to automatically place the concrete. However, it requires the pre-reserved holes to be left in the elevator shaft, which has high requirements for the construction environment and cannot adapt to dynamic construction environments. Moreover, the automatic placement of concrete using the control system and sensors without manual supervision is prone to affecting the construction quality due to environmental changes.

[0005] Method three involves using a voice system to control the construction concrete placing boom. First, the construction personnel at the control console send voice commands to the system. Then, the voice recognition module converts the voice into text and sends the text commands to the construction concrete placing boom. After receiving the commands, the construction concrete placing boom converts the text commands into executable action commands and then controls the boom to move and place the concrete according to the action commands. However, this method uses a cascaded voice-text-control voice recognition scheme for control, which carries the risk of command delay and semantic loss.

[0006] The aforementioned existing technologies have drawbacks such as high manual labor intensity, low efficiency, instruction delay and semantic loss, and limited application scenarios. The main reason for this is that the equipment modification is insufficient and the application scenarios of the device are not fully considered, which is not conducive to promotion and application. Summary of the Invention

[0007] In view of this, the purpose of this invention is to propose a construction concrete placing robot control method based on VLAS model design, which realizes the function of controlling the concrete placing robot to track the movement trajectory of construction personnel through voice commands and controlling the concrete placing robot to automatically place materials through voice commands.

[0008] To achieve the above-mentioned technical objectives, the technical solution adopted by this invention is as follows:

[0009] This invention provides a control method for a construction material placement robot based on a VLAS model, comprising the following steps:

[0010] Step 1: Receive voice commands from construction workers, verify their identities, and generate voice command embedding vectors.

[0011] Step 2: Align the voice command embedding vector with the RGB image obtained from the construction site monitoring camera in the shared semantic space, fuse visual and voice features, and generate a joint representation;

[0012] Step 3: Decompose the task based on the joint representation to obtain sub-tasks;

[0013] Step 4: Collect image data of the construction site and identify target objects, while acquiring point cloud data and extracting the 3D coordinates of the target objects;

[0014] Step 5: Generate the motion trajectory of the target object for each subtask based on the 3D coordinates of the target object, the motion control parameters of the cloth robot, and the joint positions;

[0015] Step 6: Convert the motion trajectory into multiple action commands, and drive the cloth-making robot to perform the corresponding actions according to the action commands;

[0016] Step 7: Real-time detection of the distance between the construction worker and the robotic arm of the fabric-laying robot; trigger an emergency stop if the distance exceeds the safe distance.

[0017] Step 8: Collect the real-time operating status information of the cloth-laying robot, and dynamically adjust the motion trajectory and pouring speed based on the operating status information.

[0018] Furthermore, step 1 specifically includes:

[0019] Step 11: The construction worker verbally describes a command to be executed, forming a voice instruction;

[0020] Step 12: Identify the voiceprint and intonation information of the voice command;

[0021] Step 13: Compare the voiceprint and intonation information of the voice command with the voiceprint and intonation information stored in the database. Based on the comparison results, verify whether the voice command comes from construction workers. If the comparison matches, it means that the voice command comes from construction workers, and the voice command is converted into an embedding vector. If the comparison does not match, it means that the voice command comes from non-construction workers, and no processing is performed.

[0022] Step 14: Obtain the identity information of the verified construction personnel and generate a unique personnel ID verification tag for each verified construction personnel.

[0023] Furthermore, in step 2, the visual and speech features are fused, specifically by using the attention mechanism of the Qwen-VL model to fuse the visual and speech features.

[0024] Furthermore, step 3 specifically includes:

[0025] Step 31: Receive the joint representation, which includes the speech features of the speech command and the image features of the visual scene;

[0026] Step 32: Analyze the joint representation and extract the task elements, and transform the abstract task elements into task units that can be executed by the cloth robot.

[0027] Step 33: Parameterize and time-series the task units based on the environmental state and the state of the cloth-laying robot;

[0028] Step 34: Output the sequence of multiple sub-tasks that the cloth robot can execute.

[0029] Furthermore, step 4 specifically includes:

[0030] Step 41: Collect image data of the construction site in real time through a camera, and use the YOLOv8 target detection algorithm to process the image data and identify the target object from the image data;

[0031] Step 42: Acquire point cloud data of the construction site using LiDAR, and extract the 3D coordinates of the target object from the point cloud data using a point cloud processing algorithm.

[0032] Furthermore, step 5 specifically includes:

[0033] Step 51: When in the context of a follow-up pouring scenario, the voice command describes the construction personnel with specified characteristics. The target object is the construction personnel in the construction site who match the description of the voice command. Based on the 3D coordinates of the target object, the motion control parameters of the cloth-laying robot, and the joint position, the Cubic Spline algorithm is used to generate a smooth motion trajectory of the cloth-laying tube following the target object.

[0034] Step 52: When in an automatic pouring scenario, the voice command describes the work area information with specified characteristics. The target object is the work area in the construction site that matches the description of the voice command. Based on the 3D coordinates of the target object, the motion control parameters of the placing robot, and the joint position, the Cubic Spline algorithm is used to generate the automatic pouring motion trajectory of the placing pipe covering the work area.

[0035] Specifically, in steps 51 and 52, based on the 3D coordinates of the target object, the motion control parameters of the cloth-laying robot, and the joint positions, the Cubic Spline algorithm is used to generate a smooth motion trajectory for the cloth-laying tube to follow the target object / an automatic pouring motion trajectory for the cloth-laying tube to cover the work area.

[0036] 1) Extract key points from the target object to form a series of 3D coordinate points as path points;

[0037] 2) Assign a timestamp to each path point. The initial time allocation is calculated based on the straight-line distance between two path points and the preset average speed.

[0038] 3) Cubic spline interpolation: Perform cubic spline interpolation independently on the three coordinate axes to obtain three spline curves with respect to time t;

[0039] 4) Check whether the interpolated trajectory meets the motion constraints of the cloth-laying robot. If not, adjust the time allocation and re-interpolate until the constraints are met.

[0040] 5) Output the trajectory according to the time sequence to form the motion trajectory of the cloth robot.

[0041] Furthermore, step 6 specifically includes:

[0042] Step 61: Receive the generated motion trajectory;

[0043] Step 62: Based on the kinematics and dynamics constraints of the cloth-making robot, the continuous trajectory is discretized into multiple trajectory segments;

[0044] Step 63: Generate motion information for each trajectory segment, including position, velocity, and acceleration parameters;

[0045] Step 64: Solve the inverse kinematics of the cloth robot based on the motion information and generate motion commands in the joint space;

[0046] Step 65: Send the action command to the actuator to drive the cloth-laying robot to complete the corresponding action.

[0047] Furthermore, step 7 specifically includes:

[0048] Step 71: Set the safe distance threshold;

[0049] Step 72: Real-time detection of the distance between the construction workers and the robotic arm of the fabric-laying robot;

[0050] Step 73: Determine whether the distance between the construction worker and the robotic arm of the fabric-laying robot exceeds the safe distance threshold. If yes, trigger the robotic arm to stop moving; otherwise, do not take any action.

[0051] Furthermore, step 8 specifically includes:

[0052] Step 81: During the execution process, the operating status information of the material laying robot is collected in real time and fed back. The operating status information includes the actual position, angle and moving speed of the pouring pipe.

[0053] Step 82: Compare and analyze the feedback operation status information with the preset status information. If the comparison results are consistent, no action is taken; if the comparison results are inconsistent, the motion trajectory and pouring speed are dynamically adjusted according to the preset status information until the preset status is reached.

[0054] This invention also provides a construction material placement robot control system based on a VLAS model, comprising:

[0055] The voice command module is used to receive voice commands from construction workers, verify the identity of construction workers, and generate voice command embedding vectors.

[0056] The visual-language alignment module is used to align the embedded vectors of voice commands with the RGB images obtained from construction site monitoring cameras in a shared semantic space, and to fuse visual and voice features to generate a joint representation.

[0057] The task decomposition module is used to decompose tasks based on joint representations to obtain subtasks.

[0058] The visual perception module is used to collect image data of the construction site and identify target objects, while acquiring point cloud data and extracting the 3D coordinates of the target objects;

[0059] The trajectory generation module is used to generate the motion trajectory of the target object for each subtask based on the 3D coordinates of the target object, the motion control parameters of the cloth robot, and the joint position.

[0060] The motion generation module is used to convert motion trajectories into multiple motion commands, and drive the cloth-laying robot to perform corresponding actions according to the motion commands;

[0061] The safety sensing module is used to detect the distance between the construction worker and the robotic arm of the material-laying robot in real time, and triggers an emergency stop when the safe distance is exceeded.

[0062] The trajectory adjustment module is used to collect the real-time operating status information of the fabric-laying robot and dynamically adjust the motion trajectory and pouring speed based on the operating status information.

[0063] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0064] This invention shares some similarities with Solution 3, both controlling the fabric-laying robot via voice commands. The difference lies in that this invention is based on a Vision-Language-Action (VLAS) model, while Solution 3 employs a cascaded speech-text-action speech recognition scheme, which carries the risk of command delay and semantic loss. This invention achieves direct mapping between speech and action, eliminating the need for text transcription and reducing latency. Visually, it uses an image + point cloud fusion scheme to improve the positioning accuracy of the fabric-laying robot.

[0065] 1) Saves manpower and reduces on-site construction personnel. This invention issues tasks via voice, requiring only 1-2 accompanying construction personnel on-site, or it can automatically pour concrete in a designated area, saving a significant amount of manpower.

[0066] 2) High measurement efficiency. The detection process is fully automated and has no time limit, allowing detection to be carried out during periods when no one is working, effectively improving measurement efficiency.

[0067] 3) It is highly convenient and has a wide range of applications. This invention can be controlled by remote voice commands and supports a variety of pouring modes, including follow-up pouring and automatic pouring, making it suitable for a variety of building scenarios. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is an execution flowchart of a construction material placement robot control method based on a VLAS model provided in an embodiment of the present invention.

[0070] Figure 2 This is an architecture diagram of a construction material placement robot control system based on a VLAS model, provided by an embodiment of the present invention. Detailed Implementation

[0071] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] Please see Figure 1The present invention provides a control method for a construction material placement robot based on a VLAS model, comprising the following steps:

[0073] Step 1: Receive voice commands from construction workers, verify their identities, and generate voice command embedding vectors; the purpose is to establish a human-computer interaction entry point and complete the authentication of construction workers' identities.

[0074] In this embodiment, step 1 specifically includes:

[0075] Step 11: The construction worker verbally describes a command to be executed, forming a voice instruction;

[0076] Step 12: Identify the voiceprint and intonation information of the voice command;

[0077] Step 13: Compare the voiceprint and intonation information of the voice command with the voiceprint and intonation information stored in the database. Based on the comparison results, verify whether the voice command comes from construction workers. If the comparison matches, it means that the voice command comes from construction workers, and the voice command is converted into an embedding vector. If the comparison does not match, it means that the voice command comes from non-construction workers, and no processing is performed.

[0078] Step 14: Obtain the identity information of the verified construction personnel and generate a unique personnel ID verification tag for each verified construction personnel.

[0079] Step 2: Align the embedded vector of the voice command with the RGB image obtained by the construction site monitoring camera in the shared semantic space, fuse visual and voice features, and generate a joint representation; the purpose is to establish the semantic association between the voice command and the visual scene and eliminate the semantic gap caused by modal differences.

[0080] In this embodiment, step 2, which involves fusing visual and speech features, specifically, uses the attention mechanism of the Qwen-VL model to fuse visual and speech features.

[0081] Step 3: Decompose the task based on the joint representation to obtain subtasks; the purpose is to realize the conversion of high-level instructions into executable logic.

[0082] In this embodiment, step 3 specifically includes:

[0083] Step 31: Receive the joint representation, which includes the speech features of the speech command and the image features of the visual scene;

[0084] Step 32: Analyze the joint representation and extract the task elements, and transform the abstract task elements into task units that can be executed by the cloth robot.

[0085] Step 33: Parameterize and time-series the task units based on the environmental state and the state of the cloth-laying robot;

[0086] Step 34: Output the sequence of multiple sub-tasks that the cloth robot can execute.

[0087] Step 4: Collect image data of the construction site and identify the target object, while acquiring point cloud data and extracting the 3D coordinates of the target object; the purpose is to achieve accurate positioning of the target object and provide coordinate reference for motion control.

[0088] In this embodiment, step 4 specifically includes:

[0089] Step 41: Collect image data of the construction site in real time through a camera, and use the YOLOv8 target detection algorithm to process the image data and identify the target object from the image data;

[0090] Step 42: Acquire point cloud data of the construction site using LiDAR, and extract the 3D coordinates of the target object from the point cloud data using a point cloud processing algorithm.

[0091] Step 5: Based on the 3D coordinates of the target object, the motion control parameters of the cloth robot, and the joint positions, generate the motion trajectory of the target object for each subtask; the purpose is to achieve path planning at the kinematic level, transform the abstract task into an executable path, and ensure the smoothness of the motion process.

[0092] In this embodiment, step 5 specifically includes:

[0093] Step 51: When in the context of a follow-up pouring scenario, the voice command describes the construction personnel with specified characteristics. The target object is the construction personnel in the construction site who match the description of the voice command. Based on the 3D coordinates of the target object, the motion control parameters of the cloth-laying robot, and the joint position, the Cubic Spline algorithm is used to generate a smooth motion trajectory of the cloth-laying tube following the target object.

[0094] Step 52: When in an automatic pouring scenario, the voice command describes the work area information with specified characteristics. The target object is the work area in the construction site that matches the description of the voice command. Based on the 3D coordinates of the target object, the motion control parameters of the placing robot, and the joint position, the Cubic Spline algorithm is used to generate the automatic pouring motion trajectory of the placing pipe covering the work area.

[0095] Specifically, in steps 51 and 52, based on the 3D coordinates of the target object, the motion control parameters of the cloth-laying robot, and the joint positions, the Cubic Spline algorithm is used to generate a smooth motion trajectory for the cloth-laying tube to follow the target object / an automatic pouring motion trajectory for the cloth-laying tube to cover the work area.

[0096] 1) Path point acquisition: Extract key points (such as boundary points, turning points, etc.) from the target object to form a series of 3D coordinate points (x, y, z) as path points;

[0097] 2) Time allocation: A timestamp is assigned to each path point. The initial time allocation is calculated based on the straight-line distance between two path points and the preset average speed.

[0098] 3) Cubic spline interpolation: Perform cubic spline interpolation independently on the three coordinate axes (x, y, z) to obtain three spline curves with respect to time t. Cubic spline interpolation ensures that the trajectory is continuous in position, velocity, and acceleration (C2 continuity) at the path points.

[0099] 4) Motion constraint handling: Check whether the interpolated trajectory meets the motion constraints (maximum speed, maximum acceleration) of the cloth robot. If not, adjust the time allocation (e.g., increase the time consumption) and re-interpolate until the constraints are met.

[0100] 5) Output the trajectory according to the time sequence to form the motion trajectory of the cloth robot.

[0101] Step 6: Convert the motion trajectory into multiple action commands, and drive the cloth-laying robot to perform corresponding actions according to the action commands; the purpose is to discretize the continuous trajectory into action commands so that the cloth-laying robot can perform actions.

[0102] In this embodiment, step 6 specifically includes:

[0103] Step 61: Receive the generated motion trajectory;

[0104] Step 62: Based on the kinematics and dynamics constraints of the cloth-making robot, the continuous trajectory is discretized into multiple trajectory segments;

[0105] Step 63: Generate motion information for each trajectory segment, including position, velocity, and acceleration parameters;

[0106] Step 64: Solve the inverse kinematics of the cloth robot based on the motion information and generate motion commands in the joint space;

[0107] Step 65: Send the action command to the actuator to drive the cloth-laying robot to complete the corresponding action.

[0108] Step 7: Real-time detection of the distance between the construction worker and the robotic arm of the material-laying robot; triggering an emergency stop when the safe distance is exceeded; the purpose is to achieve safety monitoring and execution assurance.

[0109] In this embodiment, step 7 specifically includes:

[0110] Step 71: Set the safe distance threshold;

[0111] Step 72: Real-time detection of the distance between the construction workers and the robotic arm of the fabric-laying robot;

[0112] Step 73: Determine whether the distance between the construction worker and the robotic arm of the fabric-laying robot exceeds the safe distance threshold. If yes, trigger the robotic arm to stop moving; otherwise, do not take any action.

[0113] Step 8: Collect the real-time operating status information of the concrete laying robot, and dynamically adjust the motion trajectory and pouring speed based on the operating status information; the purpose is to achieve closed-loop control and adaptive optimization to ensure the consistency of construction quality.

[0114] In this embodiment, step 8 specifically includes:

[0115] Step 81: During the execution process, the operating status information of the material laying robot is collected in real time and fed back. The operating status information includes the actual position, angle and moving speed of the pouring pipe.

[0116] Step 82: Compare and analyze the feedback operation status information with the preset status information. If the comparison results are consistent, no action is taken; if the comparison results are inconsistent, the motion trajectory and pouring speed are dynamically adjusted according to the preset status information until the preset status is reached.

[0117] Application scenarios include multiple modes such as follow-up pouring and automatic pouring, realizing the intelligence and automation of concrete placing machines, reducing manpower and improving economic efficiency.

[0118] Scenario 1: Dynamic casting

[0119] A control method for a construction material placement robot based on a Vision-Language-Motion (VLAS) model includes:

[0120] First, the construction worker verbally gives a voice command, such as "follow the movement of the person in the red reflective vest with the pouring pipe". The voice command module begins to recognize the command and confirms whether it is a construction worker based on information such as voiceprint and tone. After confirming that it is a construction worker based on the voice, it outputs the voice command embedding vector + personnel ID verification tag.

[0121] The voice command embedding vector and the RGB image obtained from the construction site monitoring camera are aligned in a shared semantic space. Then, the visual / voice features are fused using the Qwen-VL attention mechanism to output a joint representation.

[0122] Tasks are decomposed based on joint representations to obtain subtasks;

[0123] The construction site is captured in real time by cameras, and the YOLOv8 target detection algorithm is used to process the images and identify personnel wearing red reflective vests. Simultaneously, LiDAR and other equipment acquire point cloud data of the construction site, and point cloud processing algorithms are used to extract the 3D coordinates of the personnel in red reflective vests relative to the base of the concrete placing boom.

[0124] After obtaining the location information, based on the motion control parameters of the construction material placing robot, the position of each joint, and the actual situation of the construction site, a smooth motion trajectory of the material placing tube is generated for each subtask according to the Cubic Spline algorithm.

[0125] Based on the motion trajectory, the task of following the cloth is broken down into multiple sub-tasks, and each sub-task corresponds to one or more action instructions that the cloth placing machine can execute.

[0126] The system monitors the minimum distance between personnel and the robotic arm in real time, maintains a safe distance, and triggers an emergency stop if the safe distance is exceeded.

[0127] Upon receiving the sub-task, the actuator of the construction material placement robot is driven to perform the corresponding action;

[0128] During the execution process, the operating status information of the material-laying robot is collected in real time, such as the actual position, angle, and moving speed of the pouring pipe, and this information is fed back. The operating status information is compared and analyzed with the preset status information to dynamically adjust the trajectory and pouring speed.

[0129] Scenario 2: Automated Pouring

[0130] First, the construction worker verbally gives a voice command, such as "follow the movement of the person in the red reflective vest with the pouring pipe". The voice command module begins to recognize the command and confirms whether it is a construction worker based on information such as voiceprint and tone. After confirming that it is a construction worker based on the voice, it outputs the voice command embedding vector + personnel ID verification tag.

[0131] The speech embedding vector and the RGB image obtained from the construction site monitoring camera are aligned in a shared semantic space, and then the visual / speech features are fused using the Qwen-VL attention mechanism to output a joint representation.

[0132] Tasks are decomposed based on joint representations to obtain subtasks;

[0133] The system collects real-time images of the construction site using cameras and processes the images using the YOLOv8 target detection algorithm to identify the required work area. At the same time, devices such as LiDAR acquire point cloud data of the construction site and use point cloud processing algorithms to extract the 3D coordinates of the work area relative to the concrete placing boom base from the point cloud data.

[0134] After obtaining the location information, based on the motion control parameters of the construction material placing robot, the position of each joint, and the actual situation of the construction site, a smooth motion trajectory of the material placing tube is generated for each subtask according to the Cubic Spline algorithm.

[0135] Based on the trajectory, the task of following the cloth is broken down into multiple sub-tasks, each sub-task corresponding to one or more instructions that the cloth placing machine can execute.

[0136] If there are construction workers on site, the distance between the workers and the robotic arm will be monitored in real time to maintain a safe distance. If the safe distance is exceeded, an emergency stop will be triggered.

[0137] Upon receiving the sub-task, the actuator of the construction material placement robot is driven to perform the corresponding action.

[0138] During the execution process, the robot's operating status information, such as the actual position, angle, and moving speed of the pouring pipe, is collected in real time and fed back. The robot's trajectory and pouring speed are dynamically adjusted based on the comparison and analysis between the fed-back operating status information and the preset status information.

[0139] like Figure 2 As shown, the present invention also provides a construction material placement robot control system based on a VLAS model, comprising:

[0140] The voice command module is used to receive voice commands from construction workers, verify the identity of construction workers, and generate voice command embedding vectors.

[0141] The visual-language alignment module is used to align the embedded vectors of voice commands with the RGB images obtained from construction site monitoring cameras in a shared semantic space, and to fuse visual and voice features to generate a joint representation.

[0142] The task decomposition module is used to decompose tasks based on joint representations to obtain subtasks.

[0143] The visual perception module is used to collect image data of the construction site and identify target objects, while acquiring point cloud data and extracting the 3D coordinates of the target objects;

[0144] The trajectory generation module is used to generate the motion trajectory of the target object for each subtask based on the 3D coordinates of the target object, the motion control parameters of the cloth robot, and the joint position.

[0145] The motion generation module is used to convert motion trajectories into multiple motion commands, and drive the cloth-laying robot to perform corresponding actions according to the motion commands;

[0146] The safety sensing module is used to detect the distance between the construction worker and the robotic arm of the material-laying robot in real time, and triggers an emergency stop when the safe distance is exceeded.

[0147] The trajectory adjustment module is used to collect the real-time operating status information of the fabric-laying robot and dynamically adjust the motion trajectory and pouring speed based on the operating status information.

[0148] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A control method for a construction material placement robot based on a VLAS model, characterized in that, The steps include the following: Step 1: Receive voice commands from construction workers, verify their identities, and generate voice command embedding vectors. Step 2: Align the voice command embedding vector with the RGB image obtained from the construction site monitoring camera in the shared semantic space, fuse visual and voice features, and generate a joint representation; Step 3: Decompose the task based on the joint representation to obtain sub-tasks; Step 4: Collect image data of the construction site and identify target objects, while acquiring point cloud data and extracting the 3D coordinates of the target objects; Step 5: Generate the motion trajectory of the target object for each subtask based on the 3D coordinates of the target object, the motion control parameters of the cloth robot, and the joint positions; Step 6: Convert the motion trajectory into multiple action commands, and drive the cloth-making robot to perform the corresponding actions according to the action commands; Step 7: Real-time detection of the distance between the construction worker and the robotic arm of the fabric-laying robot; trigger an emergency stop if the distance exceeds the safe distance. Step 8: Collect the real-time operating status information of the cloth-laying robot, and dynamically adjust the motion trajectory and pouring speed based on the operating status information.

2. The construction material placement robot control method based on VLAS model design as described in claim 1, characterized in that, Step 1 specifically includes: Step 11: The construction worker verbally describes a command to be executed, forming a voice instruction; Step 12: Identify the voiceprint and intonation information of the voice command; Step 13: Compare the voiceprint and intonation information of the voice command with the voiceprint and intonation information stored in the database. Based on the comparison results, verify whether the voice command comes from construction workers. If the comparison matches, it means that the voice command comes from construction workers, and the voice command is converted into an embedding vector. If the comparison does not match, it means that the voice command comes from non-construction workers, and no processing is performed. Step 14: Obtain the identity information of the verified construction personnel and generate a unique personnel ID verification tag for each verified construction personnel.

3. The construction material placement robot control method based on VLAS model design as described in claim 1, characterized in that, Step 2 involves fusing visual and speech features, specifically by using the attention mechanism of the Qwen-VL model to fuse visual and speech features.

4. The construction material placement robot control method based on VLAS model design as described in claim 1, characterized in that, Step 3 specifically includes: Step 31: Receive the joint representation, which includes the speech features of the speech command and the image features of the visual scene; Step 32: Analyze the joint representation and extract the task elements, and transform the abstract task elements into task units that can be executed by the cloth robot. Step 33: Parameterize and time-series the task units based on the environmental state and the state of the cloth-laying robot; Step 34: Output the sequence of multiple sub-tasks that the cloth robot can execute.

5. The construction material placement robot control method based on VLAS model design as described in claim 1, characterized in that, Step 4 specifically includes: Step 41: Collect image data of the construction site in real time through a camera, and use the YOLOv8 target detection algorithm to process the image data and identify the target object from the image data; Step 42: Acquire point cloud data of the construction site using LiDAR, and extract the 3D coordinates of the target object from the point cloud data using a point cloud processing algorithm.

6. The construction material placement robot control method based on VLAS model design as described in claim 1, characterized in that, Step 5 specifically includes: Step 51: When in the context of a follow-up pouring scenario, the voice command describes the construction personnel with specified characteristics. The target object is the construction personnel in the construction site who match the description of the voice command. Based on the 3D coordinates of the target object, the motion control parameters of the cloth-laying robot, and the joint position, the Cubic Spline algorithm is used to generate a smooth motion trajectory of the cloth-laying tube following the target object. Step 52: When in an automatic pouring scenario, the voice command describes the work area information with specified characteristics. The target object is the work area in the construction site that matches the description of the voice command. Based on the 3D coordinates of the target object, the motion control parameters of the placing robot, and the joint position, the Cubic Spline algorithm is used to generate the automatic pouring motion trajectory of the placing pipe covering the work area. Specifically, in steps 51 and 52, based on the 3D coordinates of the target object, the motion control parameters of the cloth-laying robot, and the joint positions, the Cubic Spline algorithm is used to generate a smooth motion trajectory for the cloth-laying tube to follow the target object / an automatic pouring motion trajectory for the cloth-laying tube to cover the work area. 1) Extract key points from the target object to form a series of 3D coordinate points as path points; 2) Assign a timestamp to each path point. The initial time allocation is calculated based on the straight-line distance between two path points and the preset average speed. 3) Perform cubic spline interpolation independently on the three coordinate axes to obtain three spline curves with respect to time t; 4) Check whether the interpolated trajectory meets the motion constraints of the cloth-laying robot. If not, adjust the time allocation and re-interpolate until the constraints are met. 5) Output the trajectory according to the time sequence to form the motion trajectory of the cloth robot.

7. The construction material placement robot control method based on VLAS model design as described in claim 1, characterized in that, Step 6 specifically includes: Step 61: Receive the generated motion trajectory; Step 62: Based on the kinematics and dynamics constraints of the cloth-making robot, the continuous trajectory is discretized into multiple trajectory segments; Step 63: Generate motion information for each trajectory segment, including position, velocity, and acceleration parameters; Step 64: Solve the inverse kinematics of the cloth robot based on the motion information and generate motion commands in the joint space; Step 65: Send the action command to the actuator to drive the cloth-laying robot to complete the corresponding action.

8. The construction material placement robot control method based on VLAS model design as described in claim 1, characterized in that, Step 7 specifically includes: Step 71: Set the safe distance threshold; Step 72: Real-time detection of the distance between the construction workers and the robotic arm of the fabric-laying robot; Step 73: Determine whether the distance between the construction worker and the robotic arm of the fabric-laying robot exceeds the safe distance threshold. If yes, trigger the robotic arm to stop moving; otherwise, do not take any action.

9. The construction material placement robot control method based on VLAS model design as described in claim 1, characterized in that, Step 8 specifically includes: Step 81: During the execution process, the operating status information of the material laying robot is collected in real time and fed back. The operating status information includes the actual position, angle and moving speed of the pouring pipe. Step 82: Compare and analyze the feedback operation status information with the preset status information. If the comparison results are consistent, no action is taken; if the comparison results are inconsistent, the motion trajectory and pouring speed are dynamically adjusted according to the preset status information until the preset status is reached.

10. A construction material placement robot control system based on a VLAS model, characterized in that, include: The voice command module is used to receive voice commands from construction workers, verify the identity of construction workers, and generate voice command embedding vectors. The visual-language alignment module is used to align the embedded vector of voice commands with the RGB images obtained by the construction site monitoring camera in a shared semantic space, and to fuse visual and voice features to generate a joint representation. The task decomposition module is used to decompose tasks based on joint representations to obtain subtasks. The visual perception module is used to collect image data of the construction site and identify target objects, while acquiring point cloud data and extracting the 3D coordinates of the target objects; The trajectory generation module is used to generate the motion trajectory of the target object for each subtask based on the 3D coordinates of the target object, the motion control parameters of the cloth robot, and the joint position. The motion generation module is used to convert motion trajectories into multiple motion commands, and drive the cloth-laying robot to perform corresponding actions according to the motion commands; The safety sensing module is used to detect the distance between the construction worker and the robotic arm of the material-laying robot in real time, and triggers an emergency stop when the safe distance is exceeded. The trajectory adjustment module is used to collect the real-time operating status information of the laying robot and dynamically adjust the movement trajectory and pouring speed based on the operating status information.

Images