Building robot task planning and skill learning method and system

Abstract

The invention discloses a building robot task planning and skill learning method and system, and relates to the technical field of intelligent construction and building robot control, and the method comprises the steps: receiving construction target description and construction environment information through a multi-modal task-skill collaborative generation model; and generating a structured semantic vector and reasoning a construction planning graph containing a plurality of subtasks. And collecting operation behavior parameters of the building robot in each subtask according to the construction planning graph, constructing an operation sample, and inputting the operation sample into the multi-modal task-skill collaborative generation model for skill learning and action instruction generation. And in the construction execution process, various sensing data are fused for real-time judgment, and if the task execution state is inconsistent with the planning state, the multi-mode task-skill collaborative generation model adjusts an instruction based on the latest state and outputs a reasoning chain. According to the method, the deployment efficiency and task adaptability of the robot system in actual construction are improved, and a foundation is laid for man-machine cooperation and unmanned construction in the field.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent construction and construction robot control, and specifically provides a method and system for task planning and skill learning of a construction robot. Background Art

[0002] With the continuous advancement of the concept of intelligent construction, the application of construction robots in construction automation, precise execution, and safety improvement has been gradually expanded. Based on the traditional path planning and control mode, in recent years, research has gradually focused on combining artificial intelligence and multi-modal perception technologies to support the autonomous operation ability of construction robots in dynamic environments. At the same time, the development of large model pre-training technology has also brought new opportunities for task understanding and skill transfer, enabling robots to have cross-scene adaptation and natural language interaction capabilities. At present, methods such as multi-modal perception fusion, task planning modeling, and expert teaching learning have been preliminarily applied in process links such as assembly construction, masonry handling, etc., and constructing an intelligent decision-making system for construction robots has become a key technical direction for the development of construction industrialization.

[0003] Although the current construction robot system already has basic path planning and control capabilities, there are still many deficiencies in task understanding, execution generalization, and skill transfer in complex environments. Existing methods mostly rely on structured modeling and fixed template task mapping, lacking the ability to fuse and analyze unstructured data such as natural language descriptions and drawing information, and it is difficult to achieve automatic mapping from human language to machine-executable actions. In addition, skill learning generally adopts a single-dimensional trajectory fitting or action sequence playback method, resulting in the inability of the robot to generate adaptive actions based on the context when facing new tasks, new environments, or unforeseen operation disturbances, thus causing low execution efficiency or safety hazards. Especially in a dynamic construction environment, the existing technology lacks an elastic mechanism in state perception, task deviation judgment, and action reconstruction capabilities, and it is difficult to cope with real-time changing construction scenarios, lacking a dynamic adjustment and interpretation feedback mechanism based on the context reasoning chain. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed.

[0005] Therefore, the technical problems solved by the present invention are: the existing methods for task execution and control of construction robots have problems such as the lack of a unified modeling mechanism for natural language and environmental information, the skill learning method relying on teaching trajectories with poor generalization ability, and the inability to dynamically feedback and adjust state deviations during task execution, and how to achieve the intelligent understanding, action generation, and real-time reasoning and adjustment of construction robots for multi-modal tasks.

[0006] To solve the above technical problems, the present invention provides the following technical solution: A method for task planning and skill learning of a construction robot, which includes receiving a construction target description and construction environment information through a multi-modal task-skill collaborative generation model, generating a structured semantic vector, and inferring a construction plan diagram including multiple subtasks. According to the construction plan diagram, the operation behavior parameters of the construction robot in each subtask are collected, an operation sample is constructed and input into the multi-modal task-skill collaborative generation model for skill learning and action instruction generation. During the construction execution process, multiple perception data are fused for real-time judgment. If the task execution state is inconsistent with the planned state, the multi-modal task-skill collaborative generation model adjusts the instruction based on the latest state and outputs an inference chain. Constructing an operation sample and inputting it into the multi-modal task-skill collaborative generation model for skill learning and action instruction generation includes expressing the operation sample in the form of a condition-action pair through a text conversion method, inputting it into the multi-modal task-skill collaborative generation model for training, adopting a joint optimization strategy to fine-tune the parameters simultaneously in the language semantic space and the action control space, and constructing an action generation head with skill generalization ability. When the model generates each action instruction, according to the current subtask state, historical operation trajectory and execution target, the next execution action is dynamically determined through an attention mechanism.

[0007] As a preferred solution of the method for task planning and skill learning of the construction robot described in the present invention, wherein: the generation of the structured semantic vector includes parsing the construction target in the form of natural language into semantic instruction segments through an encoding mechanism, and combining the construction site map, construction drawings, known obstacle information and the initial position of the robot to generate a unified multi-dimensional tensor representation through a position mapping and semantic fusion method. The multi-dimensional tensor representation completes the alignment of different modal information through the cross-modal attention mechanism of the multi-modal task-skill collaborative generation model to form a task-driven semantic vector.

[0008] As a preferred solution of the method for task planning and skill learning of the construction robot described in the present invention, wherein: the inference of the construction plan diagram including multiple subtasks includes that the multi-modal task-skill collaborative generation model retrieves the historical task structure template in the task semantic space according to the structured semantic vector, and identifies the feasible subtask process through a graph matching mechanism. In the determined process, the types of tools required for each subtask, the operation sequence and the start and end positions in space are marked to form a construction plan diagram composed of nodes and dependency relationships. Each node represents a subtask, and each edge represents the sequential execution relationship between subtasks.

[0009] As a preferred solution of the construction robot task planning and skill learning method described in the present invention, wherein: collecting the operation behavior parameters of the construction robot in each subtask according to the construction planning drawing includes, during the execution of the specified subtask by the robot, recording the end effector position, joint angle, velocity vector and real-time force feedback information at fixed time intervals, and performing status annotation at key operation points to form time-series behavior data. The time-series behavior data is classified according to the subtask number and then standardized, and is transformed into an instruction sequence recognizable by the model through the behavior coding rule to form an operation sample.

[0010] As a preferred solution of the construction robot task planning and skill learning method described in the present invention, wherein: constructing the operation sample and inputting it into the multi-modal task-skill collaborative generation model for skill learning and action instruction generation includes, in the collected and standardized robot time-series operation sample, transforming each state-action sequence into a condition-action pair represented in text format according to the operation context, and the text content includes the semantic of the current construction task objective, the summary of the current state of the robot, and the corresponding action description under expert teaching, to form an input sequence with a language expression structure. The input sequence is used as fine-tuning corpus and input into the multi-modal task-skill collaborative generation model. The joint optimization strategy is introduced to simultaneously adjust the parameter coupling structure between the language decoder and the action decoder in the model, and the bi-directional loss function is used to optimize the language semantic space and the action control space synchronously, and an action generation head supporting the generalization of multiple types of subtasks is constructed. The action generation head calls the attention mechanism to match the optimal instruction path according to the semantic vector of the current state, historical execution trajectory and execution target of each subtask during the inference stage, and dynamically decides the next control action.

[0011] As a preferred solution of the construction robot task planning and skill learning method described in the present invention, wherein: fusing multiple perception data for real-time judgment during the construction execution process includes continuously obtaining the multi-source perception signals of the robot during the construction process, performing time synchronization and fusing them into a unified state vector. Setting a state deviation judgment threshold, when the difference between the current state and the model predicted state exceeds the set threshold, triggering the task state inconsistency flag, and inputting the current state vector into the multi-modal task-skill collaborative generation model to generate a new action output.

[0012] As a preferred solution of the construction robot task planning and skill learning method described in the present invention, wherein: the multi-modal task-skill collaborative generation model adjusts the instruction based on the latest state and outputs the inference chain includes, while the multi-modal task-skill collaborative generation model generates action instructions according to the current state, generating a traceable text description including the inference process. The text description records the inference starting point, judgment path and task decomposition basis according to the decision chain structure, and attaches the allocation result of each subtask and the corresponding triggering logic condition.

[0013] Another object of the present invention is to provide a construction robot task planning and skill learning system, which can generate an instruction module through learning skills, solve the problems existing in the existing construction robot task execution and control methods, such as the lack of a unified modeling mechanism for natural language and environmental information, the poor generalization ability of the skill learning method relying on teaching trajectories, and the inability to dynamically feedback and adjust the state deviation during task execution, as well as the problems of how to realize the intelligent understanding, action generation and real-time reasoning and adjustment of multi-modal tasks by construction robots.

[0014] As a preferred solution of the construction robot task planning and skill learning system described in the present invention, it includes a parsing target generation planning diagram module, a learning skill generation instruction module, and a judgment deviation dynamic adjustment module.

[0015] The parsing target generation planning diagram module is used to receive the construction target description and construction environment information through a multi-modal task-skill collaborative generation model, generate a structured semantic vector and infer a construction planning diagram including multiple subtasks.

[0016] The learning skill generation instruction module is used to collect the operation behavior parameters of the construction robot in each subtask according to the construction planning diagram, construct an operation sample and input it into the multi-modal task-skill collaborative generation model for skill learning and action instruction generation.

[0017] The judgment deviation dynamic adjustment module is used to perform real-time judgment by integrating various perception data during the construction execution process. If the task execution state is inconsistent with the planned state, the multi-modal task-skill collaborative generation model adjusts the instruction based on the latest state and outputs an inference chain.

[0018] A computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the construction robot task planning and skill learning method.

[0019] A computer-readable storage medium stores a computer program thereon. When the computer program is executed by a processor, it implements the steps of the construction robot task planning and skill learning method.

[0020] Advantages of the present invention: The method for task planning and skill learning of a construction robot provided by the present invention inputs the construction task objectives and environmental information into a multi-modal task-skill collaborative generation model, and combines data such as natural language semantic parsing, drawing structuring, obstacle information annotation, and robot pose to generate a unified structured semantic vector and infer a construction plan diagram, realizing the cross-modal fusion of natural language objectives, construction drawing semantics, and environmental perception, and constructing a unified expression form among multi-source heterogeneous data. It endows the construction robot with the ability to "understand construction semantics and autonomously plan subtasks", breaking through the limitation of the traditional method that can only drive the robot relying on structured process template instructions, and enabling the system to have stronger task generalization and autonomous planning capabilities.

[0021] By collecting the execution behavior parameters of the construction robot in each subtask, including multi-dimensional data such as position, angle, speed, and force, and standardizing and encoding them according to the time series, converting them into structured "condition-action pair" text statements, and inputting them into the large model for fine-tuning training, it achieves the effect of significantly improving the skill transfer ability and action generation ability of the robot in new tasks and complex environments, solves the problems of the traditional trajectory learning being unable to generalize and having a strong dependence on teaching, and reduces the sample requirements and labor costs for model updating.

[0022] By real-time fusing multi-source perception data such as images, force sense, IMU, and voice of the robot during construction, constructing the current state vector and judging the deviation from the expected state, when the state is abnormal, triggering the update mechanism and having the model regenerate instructions and inference chains, it achieves the effect that the system can realize a self-feedback closed-loop control mechanism with high robustness and low error rate during construction, significantly enhancing the flexibility and safety during task execution, and breaking through the technical bottleneck that the existing technology cannot adaptively adjust the robot under state fluctuations. Description of the Drawings

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0024] Figure 1 It is the overall flowchart of a method for task planning and skill learning of a construction robot provided by the first embodiment of the present invention. Detailed Embodiments

[0025] To make the above objects, features, and advantages of the present invention more apparent and understandable, the following provides a detailed description of the specific embodiments of the present invention in conjunction with the accompanying drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0026] Embodiment 1, referring to Figure 1 , which is an embodiment of the present invention, provides a method for task planning and skill learning of a construction robot, including: S1: Receive a construction target description and construction environment information through a multi-modal task-skill collaborative generation model, generate a structured semantic vector, and infer a construction plan graph including multiple subtasks.

[0027] Parse the construction target in the form of natural language into semantic instruction fragments through an encoding mechanism, and combine the construction site map, construction drawings, known obstacle information, and the initial position of the robot to generate a unified multi-dimensional tensor representation through position mapping and semantic fusion. The tensor representation completes the alignment of different modal information through the cross-modal attention mechanism of the multi-modal task-skill collaborative generation model to form a task-driven semantic vector.

[0028] The multi-modal task-skill collaborative generation model retrieves historical task structure templates in the task semantic space according to the structured semantic vector, and identifies a feasible subtask process through a graph matching mechanism. Mark the types of tools required for each subtask, the operation sequence, and the start and end positions in space in the determined process to form a construction plan graph composed of nodes and dependency relationships. Each node represents a subtask, and each edge represents the sequential execution relationship between subtasks.

[0029] A preferred solution for parsing construction objectives in natural language form into semantic instruction fragments through an encoding mechanism specifically includes the following steps. First, semantic analysis is performed on the construction objectives described in natural language. A large-scale pre-trained model based on the Transformer architecture is used to extract semantic components such as key verbs, target objects, and action sequences in the task, and convert them into an intermediate representation of task instruction fragments. At the same time, the construction site map and construction drawing data are vectorized, and coordinate alignment is performed with obstacle information through a spatial registration method to obtain an environmental topological structure. The above semantic instructions and environmental information are fused through a position mapping mechanism to form a unified multi-dimensional tensor representation, which is used to describe the "objective-site-path" triple relationship in the current task scenario. This multi-dimensional tensor is then input into a multi-modal task-skill collaborative generation model. Inside the model, cross-modal attention mechanisms are used to align language semantics and environmental structures, enabling the model to understand the relevance of various inputs in a shared semantic space. This process enables the large model to obtain a task-driven semantic vector, representing the optimal task expression in the current construction objective and environmental state.

[0030] A preferred solution for the multi-modal task-skill collaborative generation model to retrieve historical task structure templates in the task semantic space based on the structured semantic vector specifically includes the following steps. Based on the semantic vector, the multi-modal task-skill collaborative generation model enters the task reasoning stage. The model retrieves historical task structure templates (such as standard operation processes like wall masonry and steel bar binding) in the task semantic space, and uses semantic similarity and graph structure matching algorithms to compare the structural relationship between the current task and the known templates, thereby identifying adaptable task processes. After successful matching, the model automatically completes task decomposition, divides the overall construction objective into several sub-tasks, and labels key execution parameters such as the required execution tool type, operation sequence, spatial start and end positions for each sub-task. Finally, all the identified and parsed sub-tasks and their execution dependencies are organized into a construction plan graph, which is represented in a graph structure. Nodes represent individual sub-task units, and edges represent the dependency order and triggering logic between tasks. This plan graph serves as the basic input for subsequent robot skill invocation and execution scheduling, ensuring that construction robots can efficiently complete task decomposition and instruction scheduling in complex scenarios.

[0031] Furthermore, a preferred solution for the multi-modal task-skill collaborative generation model specifically includes that the multi-modal task-skill collaborative generation model consists of three parts: a semantic encoder, an environmental fusion encoder, and a task graph decoder. The three are linked through cross-modal attention mechanisms to collaboratively complete semantic understanding and task structure generation.

[0032] The semantic encoder is used to perform in-depth semantic analysis on the input natural language of construction objectives. This part adopts a multi-layer cascaded language encoding unit structure, and each layer contains word vector embedding, context-dependent modeling, and attention mechanism to extract key semantic components such as verbs, objects, and operation instructions in the text. Finally, a unified semantic vector is formed as the semantic representation of the task objective.

[0033] The environment fusion encoder is mainly responsible for uniformly encoding construction drawings, obstacle information, and the initial pose of the robot. First, the drawing structure is transformed into a structure vector through graph structure embedding technology, the obstacle information is input into the sparse feature extraction network in the form of spatial point clouds for spatial compression, and the initial pose is directly mapped into a six-dimensional space representation. After the above multi-source data passes through the dimension alignment and fusion module, a set of fusion vectors representing the current construction environment state is output.

[0034] The task graph decoder receives the outputs from the semantic encoder and the environment fusion encoder. It internally uses a cross-modal attention mechanism to interactively fuse semantic and spatial information, and combines historical task templates for structural reasoning. The model first performs weighted matching on the semantic and environment vectors, then locates the closest historical task structure through the template retrieval mechanism, and dynamically adjusts the sub-task nodes and execution relationships according to the current semantic intention. Finally, a task planning graph is output, where each node in the graph represents an executable sub-task, and each edge represents the dependency order and logical connection between sub-tasks. This structure diagram provides clear task boundaries and context constraints for subsequent action generation and skill learning.

[0035] Each part in the above model structure forms a closed-loop interaction. The semantic encoder provides semantic objectives, the environment encoder provides spatial conditions, and the decoder fuses the two to generate an operable task structure diagram. The model has the ability to jointly understand task language and complex construction environments, and supports flexible structure matching and dynamic task decomposition to ensure that the robot can generate a task path diagram with execution logic based on multi-modal inputs.

[0036] Furthermore, the model structure design fully considers the heterogeneity and modal differences of the input data. Traditional neural network models are mostly constructed for a single modality, but the semantic encoder, environment fusion encoder, and trajectory behavior module adopted in the present invention perform dedicated modeling for different types of data such as natural language, graph structure, spatial point sets, and poses. After independent encoding of each channel, the representation learning of the shared semantic space is completed through the cross-modal attention mechanism, ensuring that heterogeneous data can be fused and input into a unified decoder for task graph prediction or action generation.

[0037] It should be noted that through the semantic parsing and multi-modal fusion mechanism, S1 uniformly encodes the natural language task objectives and environmental information, constructs a task-driven semantic vector, and generates a construction plan diagram, realizing the automatic conversion from human language to robot-executable tasks, improving the accuracy of task understanding and the intelligence of planning, and ensuring that the robot can efficiently adapt to complex scenarios.

[0038] S2: Collect the operation behavior parameters of the construction robot in each subtask according to the construction plan diagram, construct operation samples, and input them into the multi-modal task-skill collaborative generation model for skill learning and action instruction generation.

[0039] During the execution of the specified subtask by the robot, record the position of the end effector, joint angles, velocity vector, and real-time force feedback information at fixed time intervals, and perform status annotation at key operation points to form time-series behavior data.

[0040] The definition of the behavior acquisition tensor is expressed as: ; Among them, the behavior tensor represents the complete behavior stream of the robot numbered in the subtask numbered , and represent the task execution time period. represents the three-dimensional velocity vector. is the joint angle gradient change tensor. is the position state function of the end effector. is the force feedback intensity collected by the sensor, is the unit impedance function, indicating the current response ability of the actuator to external disturbances.

[0041] is expressed as: ; Among them, represents the initial displacement. represents the current speed of the robot.

[0042] is expressed as: ; Among them, represents the force feedback intensity collected by the sensor at the time of . represents the robot 's impedance adjustment coefficient.

[0043] The time-series behavior data is classified according to the subtask number and then standardized, expressed as: ; where is the standardized behavior sample sequence numbered , represents the change rate of the th joint unit in the sample, represents the total number of frames in the sample.

[0044] It is transformed into an instruction sequence recognizable by the model through the behavior encoding rule to form an operation sample. In the collected and standardized robot time-series operation samples, each state-action sequence is transformed into a condition-action pair represented in text format according to the operation context. The text content includes the semantic of the current construction task objective, the summary of the current state of the robot, and the corresponding action description under expert teaching, forming an input sequence with a language expression structure, expressed as: ; where is the finally generated text sentence structure, represents the formatting function, is the semantic vector of the construction objective, is the Sigmoid filtering value of the standardized behavior sequence, is the structure description vector of the current action under expert teaching.

[0045] The input sequence is used as fine-tuning corpus and input into the multi-modal task-skill collaborative generation model. Further, the training process of the multi-modal task-skill collaborative generation model includes that in the training stage, the natural language description of the construction task is first input into the semantic encoder. During the training process, the encoder extracts semantic instructions, target entities, and operation behaviors in the sentence layer by layer through multi-round context-related modeling, and finally outputs a unified semantic embedding vector. This vector will be used as the basic expression for the model to understand the task objective during training.

[0046] Subsequently, the construction drawing information, the set of obstacle points, and the initial pose of the robot are used as environmental inputs and processed by the environmental fusion encoder. The drawing structure is vectorized and encoded using a graph neural network, the obstacle information uses a sparse point processing network to extract spatial features, and the robot pose is directly mapped to a spatial state representation. The three are integrated into a unified environmental representation through a fusion mechanism to provide spatial constraint conditions.

[0047] The semantic vector and the environmental vector are simultaneously input into the task graph decoder. During the training process, the decoder uses the internal attention matching mechanism to adjust the generation result in combination with the real task structure label. The system compares with the manually annotated task structure diagram, and optimizes the accuracy of node generation and the rationality of the task process layer by layer. The training objective is to enable the model to accurately restore the sub-task decomposition structure and execution order of the real construction task.

[0048] The model training is optimized using a phased structure loss function, and the classification accuracy of task nodes and the structural connection relationship between subtasks are independently supervised. Among them, the node recognition part evaluates whether the model correctly identifies the types and orders of each subtask, while the structural relationship part judges the topological consistency between the task dependency graph generated by the model and the manual task graph. In addition, to enhance the robustness of the model, a regularization term is added to limit overfitting.

[0049] The entire training process adopts a batch gradient optimization strategy. In each round of training, the internal parameters of the semantic encoding layer, the environment encoding layer, and the task graph decoding layer are adjusted through an error feedback mechanism to achieve the collaborative optimization of multi-modal information fusion and structured task generation. After training, the model can automatically output a construction subtask flow chart with reasonable structure and clear logic based on natural language goals and environmental states under new task inputs.

[0050] A joint optimization strategy is introduced to simultaneously adjust the parameter coupling structure between the language decoder and the action decoder in the model. The language semantic space and the action control space are tuned synchronously through a bidirectional loss function to construct an action generation head that supports the generalization of multiple types of subtasks. The action generation head calls the attention mechanism to match the optimal instruction path according to the semantic vectors of the current state, historical execution trajectory, and execution target of each subtask during the inference phase, and dynamically decides the next control action. The action generation vector reasoning is expressed as: ; where represents the action control vector output for subtask at time . represents the historical trajectory backtracking function. is the target guiding function. is the current state representation function. is the action encoding function.

[0051] is expressed as: ; where represents the historical trajectory window length. is the trajectory state vector of the robot at time .

[0052] is expressed as: ; where represents the target position vector. Represents the current estimated position vector of the robot. Represents the target error adjustment coefficient.

[0053] Is expressed as: ; Wherein, Represents the dimension of the current state vector of the robot. Represents the value of the current state variable of the robot in the

[0054] A preferred scheme of the bidirectional loss function specifically includes that the language-action joint optimization loss function is expressed as: ; Wherein, is the joint loss function. is the number of sample batches. and are the original and predicted semantic decoding vectors respectively. and are the action space output and prediction distribution. Represents the KL divergence metric, and are the sample feature complexity adjustment factors.

[0055] It should be noted that S2 accurately captures the dynamic execution characteristics of the construction robot in each subtask by constructing a behavior acquisition tensor, and introduces a standardization and texturization mechanism to convert low-level control signals into a language expression structure that can be understood by the multimodal generation model, establishing a triple alignment mechanism of "state-semantics-action". Compared with the existing learning methods that rely solely on pure teaching trajectories or vector instructions, this step realizes cross-modal transfer training and language space generalization, and significantly improves the action instruction generation ability and response speed under new tasks by adjusting the parameter coupling of the language and action spaces through a joint optimization mechanism. This method breaks through the technical bottlenecks of traditional robot models with poor transfer ability and high teaching costs in new scenarios, and has stronger scalability and intelligent adaptation ability.

[0056] S3: During the construction execution process, fuse various perception data for real-time judgment. If the task execution state is inconsistent with the planned state, the multimodal task-skill collaborative generation model adjusts the instruction based on the latest state and outputs an inference chain.

[0057] Continuously obtain the multi-source perception signals of the robot during the construction process, perform time synchronization and fuse them into a unified state vector. Set a state deviation judgment threshold. When the difference between the current state and the model prediction state exceeds the set threshold, trigger the task state inconsistency flag, and input the current state vector into the multimodal task-skill collaborative generation model to generate a new action output.

[0058] Further, during the construction task execution of the construction robot, the system continuously collects, fuses, and performs real-time status judgment on multi-source perception signals to achieve dynamic task execution regulation and model feedback update. The perception data includes, but is not limited to: visual image stream, force sensor output, inertial measurement unit (IMU) data, depth camera point cloud information, and voice interaction input. All data is aligned according to the timestamp synchronization mechanism to construct a unified time-series status vector.

[0059] This status vector undergoes status fusion at a fixed period (every 50 ms). After modal alignment and feature compression, a current status representation vector with a unified dimension is formed. Meanwhile, the expected status vector inferred by the multi-modal task-skill collaborative generation model based on the task planning diagram is retained in the system.

[0060] To determine whether the current execution status of the robot deviates from the expected planned path, the system sets the status deviation judgment threshold to 0.12, which is determined according to the empirical optimal solution during the model tuning process. The system calculates the Euclidean distance between the current status and the predicted status in each cycle: ; where represents the current status representation vector with a unified dimension, represents the expected status vector.

[0061] If is greater than 0.12, it is regarded that the task execution status is inconsistent with the planned status, and the status update process is triggered. The status update process includes freezing the current execution thread and saving the on-site data. The latest status vector is input into the model, and combined with the historical trajectory and target semantics, new action instructions are regenerated. An inference chain containing the adjustment basis is output, and the planning diagram is updated.

[0062] At this time will be re-transmitted as the latest status input to the multi-modal task-skill collaborative generation model to trigger the immediate inference process.

[0063] In this process, the model combines the current task semantics, historical execution records, and sub-task context status to regenerate the next action instruction and simultaneously generates a traceable inference chain text description. The inference chain is represented in a tree structure and includes the following elements:

[0064] Inference starting point: Summary of the current status input.

[0065] Judgment path: The semantic backtracking and policy matching path inside the model (such as "the current status deviates from the target trajectory direction → detect abnormal load of the execution arm → adjust the path").

[0066] Basis for task decomposition: Specify how the model re - decomposes tasks according to templates in the task library (such as "sub - task structure for obstacle avoidance").

[0067] Sub - task assignment result: Update the execution responsibility assignment and estimated time consumption for each task node.

[0068] Trigger logic condition: Mark the status judgment as "active reconstruction" or "passive repair", and record the triggered threshold and execution timestamp.

[0069] While the multi - modal task - skill collaborative generation model generates action instructions according to the current state, it generates a traceable text description containing the reasoning process. The text description records the reasoning starting point, judgment path, and task decomposition basis according to the decision - chain structure, and attaches the assignment result of each sub - task and the corresponding trigger logic condition.

[0070] Furthermore, after the inference chain is output, it is synchronously sent to the control terminal and the supervision system, and automatically archived in the model iterative training pool as incremental data samples for the subsequent fine - tuning and reinforcement learning stages.

[0071] In the present invention, the image state deviation trigger threshold is set to 0.08. The force feedback abnormal threshold is set to 3.5N. The end - position drift threshold is set to 5mm. The IMU angular velocity change rate threshold is set to 0.5rad / s.

[0072] It should be noted that S3 dynamically adjusts the robot's execution strategy by fusing multi - source perception data in real - time and comparing it with the expected task planning state. When a deviation is detected, an inference chain is immediately generated and the action instructions are optimized to ensure that the robot can flexibly adapt in a complex environment. Through the traceability of the inference chain, the interpretability and fault - tolerance of task execution are improved, ensuring efficient and accurate construction control and overcoming the problem of poor adaptability of traditional methods in dynamic environments.

[0073] Embodiment 2, an embodiment of the present invention, provides a construction robot task planning and skill learning system, including a parsing target generating a planning diagram module, a learning skill generating instruction module, and a judging deviation dynamic adjustment module.

[0074] The parsing target generating a planning diagram module is used to receive the construction target description and construction environment information through the multi - modal task - skill collaborative generation model, generate a structured semantic vector, and infer a construction planning diagram containing multiple sub - tasks.

[0075] The parsing target generating a planning diagram module includes a semantic parsing sub - module and a task planning diagram generating sub - module.

[0076] Furthermore, the semantic parsing sub-module is used to receive the construction task objectives in the form of natural language, encode the target semantics into semantic vectors through the Transformer architecture model, and generate a unified input tensor in combination with information such as construction drawings, site topologies, and the initial poses of robots. The task planning diagram generation sub-module is used to infer a construction plan diagram containing multiple sub-tasks based on the input tensor. The plan diagram represents the execution dependency relationships between nodes in the form of a graph structure. Each node corresponds to an atomic construction operation, and the edges represent the task sequence and logical conditions.

[0077] It should be noted that the semantic parsing sub-module is the starting point of the parsing target generation plan diagram module, and the semantic representation output by it serves as the input basis for subsequent sub-task inferences. The task planning diagram generation sub-module completes the structural expression of the current construction task based on historical task templates and structure matching mechanisms, providing a task framework with clear logic for the skill learning stage.

[0078] It should also be noted that the parsing target generation plan diagram module, as the information entry point of this system, completes the cross-modal conversion from natural language targets to structured task diagrams, which is the basis of the intelligent planning ability of the present invention.

[0079] The learning skill generation instruction module is used to collect the operation behavior parameters of the construction robot in each sub-task according to the construction plan diagram, construct operation samples and input them into the multi-modal task-skill collaborative generation model for skill learning and action instruction generation.

[0080] The learning skill generation instruction module includes a behavior collection and encoding sub-module and an action instruction generation sub-module.

[0081] Furthermore, the behavior collection and encoding sub-module is used to collect behavior parameters such as position, angle, and force feedback during the execution of sub-tasks by the robot, perform normalization and time series alignment processing on them, and then encode them into standardized "condition-action pairs" according to the task context structure. The action instruction generation sub-module is used to input the standardized behavior samples into the multi-modal task-skill collaborative generation model, and perform training and fine-tuning through the semantic-action space collaborative optimization mechanism, so as to generate action instruction outputs adapted to various sub-tasks.

[0082] It should be noted that the behavior collection and encoding sub-module not only ensures the structural consistency of the original operation sample data, but also provides training corpus with semantic relevance for model fine-tuning. The action instruction generation sub-module, as the core module of skill transfer, supports the generation of generalization execution strategies for untaught tasks.

[0083] It should also be noted that the learning skill generation instruction module plays the role of "from experience to strategy" in the present invention, enabling the robot to have the ability of continuous learning and dynamic adaptation when facing different construction tasks, which is the key link to realize intelligent skill transfer.

[0084] The judgment deviation dynamic adjustment module is used to perform real-time judgment by integrating various perception data during the construction execution. If the task execution status is inconsistent with the planned status, the multi-modal task-skill collaboration generation model adjusts the instruction based on the latest status and outputs the inference chain.

[0085] The judgment deviation dynamic adjustment module includes a status fusion judgment sub-module and an inference chain generation sub-module.

[0086] Furthermore, the status fusion judgment sub-module is used to collect multi-modal perception data (such as visual images, IMU postures, force sensor information, etc.) in real time during the construction process, perform time synchronization and feature fusion, construct a unified state vector, and compare it with the expected state. If the deviation is greater than the set threshold, the status inconsistency flag is triggered. The inference chain generation sub-module is used to re-enter the current state into the multi-modal generation model when the status deviation occurs, generate new action instructions and synchronously output the traceable inference chain text. The inference chain records the task reconstruction path, judgment basis, and update strategy.

[0087] It should be noted that the status fusion judgment sub-module ensures that the robot has the ability to perceive and identify deviations in real time for sudden changes in the complex construction environment. The inference chain generation sub-module enhances the interpretability of the model decision-making process and provides an interface for manual intervention and a source of incremental training samples.

[0088] It should also be noted that the judgment deviation dynamic adjustment module implements a "status deviation - action reconstruction - logical traceability" closed-loop feedback mechanism, which is a guarantee mechanism for the present invention to support high-robustness task execution, and significantly improves the environmental adaptability and execution flexibility of the system.

Claims

1. A method for task planning and skill learning of a construction robot, characterized in that, Including: Receiving a construction target description and construction environment information through a multi-modal task-skill collaborative generation model, generating a structured semantic vector, and inferring a construction plan diagram containing multiple subtasks; Collecting the operation behavior parameters of the construction robot in each subtask according to the construction plan diagram, constructing an operation sample, and inputting it into the multi-modal task-skill collaborative generation model for skill learning and action instruction generation; During the construction execution process, fusing various perception data for real-time judgment. If the task execution state is inconsistent with the planned state, the multi-modal task-skill collaborative generation model adjusts the instruction based on the latest state and outputs an inference chain; Constructing an operation sample and inputting it into the multi-modal task-skill collaborative generation model for skill learning and action instruction generation includes expressing the operation sample in the form of a condition-action pair through a text conversion method, inputting it into the multi-modal task-skill collaborative generation model for training, using a joint optimization strategy to fine-tune the parameters simultaneously in the language semantic space and the action control space, and constructing an action generation head with skill generalization ability; when the model generates each action instruction, according to the current subtask state, historical operation trajectory, and execution target, dynamically determine the next execution action through the attention mechanism.

2. The method for task planning and skill learning of a construction robot according to claim 1, wherein: The generating of the structured semantic vector includes: Parsing the construction target in natural language form into semantic instruction fragments through an encoding mechanism, and combining the construction site map, construction drawings, known obstacle information, and the initial position of the robot to generate a unified multi-dimensional tensor representation through position mapping and semantic fusion; The multi-dimensional tensor representation completes the alignment of different modal information through the cross-modal attention mechanism of the multi-modal task-skill collaborative generation model to form a task-driven semantic vector.

3. The method for task planning and skill learning of a construction robot according to claim 1 or 2, characterized in that: The inferring of the construction plan diagram containing multiple subtasks includes: The multi-modal task-skill collaborative generation model retrieves the historical task structure template in the task semantic space according to the structured semantic vector, and identifies the feasible subtask process through the graph matching mechanism; Mark the types of tools required for each subtask, the operation sequence, and the starting and ending positions in space in the determined process to form a construction plan diagram composed of nodes and dependency relationships. Each node represents a subtask, and each edge represents the sequential execution relationship between subtasks.

4. The method for task planning and skill learning of a construction robot according to claim 3, characterized in that: The collecting of the operation behavior parameters of the construction robot in each subtask according to the construction plan diagram includes: During the robot's execution of the specified subtask, record the end effector position, joint angle, velocity vector, and real-time force feedback information at fixed time intervals, and perform status marking at key operation points to form time-series behavior data; The time-series behavior data is classified according to the subtask number and then standardized, and is transformed into an instruction sequence recognizable by the model through the behavior encoding rule to form an operation sample.

5. The method for building robot task planning and skill learning according to any one of claims 1, 2 or 4, characterized in that: The constructing of an operation sample and inputting it into the multi-modal task-skill collaborative generation model for skill learning and action instruction generation includes: In the collected and standardized robot sequential operation samples, each state-action sequence is transformed into a condition-action pair represented in text format according to the operation context. The text content includes the semantic of the current construction task objective, the summary of the current robot state, and the corresponding action description under expert teaching, forming an input sequence with a language expression structure; The input sequence is used as fine-tuning corpus and input into the multi-modal task-skill collaborative generation model. A joint optimization strategy is introduced to simultaneously adjust the parameter coupling structure between the language decoder and the action decoder in the model. Through a bidirectional loss function, the language semantic space and the action control space are optimized synchronously, and an action generation head that supports generalization of multiple types of subtasks is constructed; During the inference phase, the action generation head calls the attention mechanism to match the optimal instruction path based on the semantic vectors of the current state, historical execution trajectory, and execution target of each subtask, and dynamically decides the next control action.

6. The method for task planning and skill learning of a construction robot according to claim 5, characterized in that: The real-time judgment by integrating multiple perception data during the construction execution process includes, During the construction process, continuously obtain the multi-source perception signals of the robot, perform time synchronization and integrate them into a unified state vector; Set a state deviation judgment threshold. When the difference between the current state and the model-predicted state exceeds the set threshold, trigger a task state inconsistency flag, and input the current state vector into the multi-modal task-skill collaborative generation model to generate a new action output.

7. The method for task planning and skill learning of a construction robot according to any one of claims 1, 2, 4 or 6, characterized in that: The adjustment of the instruction by the multi-modal task-skill collaborative generation model based on the latest state and the output of the inference chain include, While the multi-modal task-skill collaborative generation model generates an action instruction according to the current state, generate a traceable text description including the inference process; The text description records the inference starting point, judgment path, and task decomposition basis according to the decision chain structure, and attaches the allocation result of each subtask and the corresponding triggering logic condition.

8. An architectural robot task planning and skill learning system, characterized in that: It includes a parsing target to generate a planning diagram module, a learning skill to generate an instruction module, and a judgment deviation dynamic adjustment module; The parsing target to generate a planning diagram module is used to receive the construction target description and construction environment information through the multi-modal task-skill collaborative generation model, generate a structured semantic vector, and infer a construction planning diagram including multiple subtasks; The learning skill to generate an instruction module is used to collect the operation behavior parameters of the construction robot in each subtask according to the construction planning diagram, construct operation samples, and input them into the multi-modal task-skill collaborative generation model for skill learning and action instruction generation; The judgment deviation dynamic adjustment module is used to perform real-time judgment by integrating multiple perception data during the construction execution process. If the task execution state is inconsistent with the planned state, the multi-modal task-skill collaborative generation model adjusts the instruction based on the latest state and outputs the inference chain.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the construction robot task planning and skill learning method described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the construction robot task planning and skill learning method described in any one of claims 1 to 7.

Images