A construction robot task planning and skill learning method and system

Abstract

The present invention discloses a method and system for task planning and skill learning of a construction robot, which relates to the field of intelligent construction and construction robot control technology, including receiving construction target descriptions and construction environment information through a multimodal task-skill collaborative generation model, generating a structured semantic vector and inferring a construction planning diagram containing multiple subtasks. According to the construction planning diagram, the operating behavior parameters of the construction robot in each subtask are collected, and operation samples are constructed and input into the multimodal task-skill collaborative generation model for skill learning and action instruction generation. During the construction execution process, multiple perception data are integrated for real-time judgment. If the task execution status is inconsistent with the planning status, the multimodal task-skill collaborative generation model adjusts the instructions based on the latest status and outputs the reasoning chain. The method described in the present invention improves the deployment efficiency and task adaptability of the robot system in actual construction, and lays the foundation for human-machine collaboration and unmanned construction in this field.

Description

Technical Field

[0001] The present invention relates to the field of intelligent construction and construction robot control technology, and specifically to a construction robot task planning and skill learning method and system. Background Art

[0002] With the continuous advancement of the concept of intelligent construction, the application of construction robots in construction automation, precision execution, and safety improvement has gradually expanded. Based on traditional path planning and control models, research in recent years has gradually focused on combining artificial intelligence with multimodal perception technology to support the autonomous operation capabilities of construction robots in dynamic environments. At the same time, the development of large-scale model pre-training technology has also brought new opportunities for task understanding and skill transfer, enabling robots to adapt across scenarios and interact with natural language. At this stage, methods such as multimodal perception fusion, task planning modeling, and expert teaching and learning have been initially applied in process links such as assembly construction and masonry and transportation. The construction of an intelligent decision-making system for construction robots has become a key technical direction for the development of construction industrialization.

[0003] Although current construction robot systems already possess basic path planning and control capabilities, they still have many deficiencies in task understanding, execution generalization, and skill transfer in complex environments. Existing methods mostly rely on structured modeling and fixed template task mapping, lacking the ability to integrate and parse unstructured data such as natural language descriptions and drawing information, making it difficult to achieve automatic mapping from human language to machine-executable actions. In addition, skill learning generally adopts a single-dimensional trajectory fitting or action sequence playback method, resulting in the robot's inability to generate adaptive actions based on context when faced with new tasks, new environments, or unforeseen operational disturbances, resulting in low execution efficiency or safety hazards. Especially in dynamic construction environments, existing technologies lack flexible mechanisms in state perception, task deviation judgment, and action reconstruction capabilities, making it difficult to cope with real-time changing construction scenarios and lacking a dynamic adjustment and interpretation feedback mechanism based on the contextual reasoning chain. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problems solved by the present invention are: the existing construction robot task execution and control methods lack a unified modeling mechanism for natural language and environmental information, the skill learning method relies on the teaching trajectory with poor generalization ability, and the state deviation during task execution cannot be dynamically fed back and adjusted, as well as how to realize the construction robot's intelligent understanding of multimodal tasks, action generation and real-time reasoning and adjustment.

[0006] To address the aforementioned technical issues, the present invention provides the following technical solutions: a method for task planning and skill learning for construction robots, comprising receiving a construction target description and construction environment information through a multimodal task-skill collaborative generation model, generating a structured semantic vector, and inferring a construction plan diagram containing multiple subtasks. Based on the construction plan diagram, the construction robot's operational behavior parameters for each subtask are collected, and operation samples are constructed and input into the multimodal task-skill collaborative generation model for skill learning and action command generation. During the construction execution process, multiple sensory data are integrated for real-time judgment. If the task execution status is inconsistent with the planned status, the multimodal task-skill collaborative generation model adjusts the instructions based on the latest status and outputs an inference chain. Constructing operation samples and inputting them into the multimodal task-skill collaborative generation model for skill learning and action command generation includes expressing the operation samples as condition-action pairs through textual conversion, inputting them into the multimodal task-skill collaborative generation model for training, and employing a joint optimization strategy to simultaneously fine-tune parameters in both the language semantic space and the action control space to construct an action generation head with skill generalization capabilities. When the model generates each action instruction, it dynamically determines the next action to be executed through the attention mechanism based on the current subtask status, historical operation trajectory and execution goal.

[0007] As a preferred embodiment of the construction robot task planning and skill learning method described in the present invention, generating a structured semantic vector involves parsing construction objectives in natural language into semantic instruction fragments through an encoding mechanism. This is then combined with a construction site map, construction drawings, known obstacle information, and the robot's initial position to generate a unified multidimensional tensor representation through position mapping and semantic fusion. This multidimensional tensor representation aligns information from different modalities through the cross-modal attention mechanism of a multimodal task-skill collaborative generation model, forming a task-driven semantic vector.

[0008] As a preferred embodiment of the construction robot task planning and skill learning method described herein, inferring a construction plan diagram containing multiple subtasks includes using a multimodal task-skill collaborative generation model to retrieve historical task structure templates in the task semantic space based on structured semantic vectors and identifying feasible subtask processes through a graph matching mechanism. Within the identified process, the required tool type, operation sequence, and spatial start and end locations of each subtask are annotated to form a construction plan diagram consisting of nodes and dependency relationships. Each node represents a subtask, and each edge indicates the sequential execution relationship between subtasks.

[0009] As a preferred embodiment of the construction robot task planning and skill learning method described herein, the method of collecting the construction robot's operational behavior parameters for each subtask based on the construction plan includes recording the end effector position, joint angles, velocity vector, and real-time force feedback information at fixed intervals while the robot performs the designated subtask, and annotating key operational points to form time-series behavioral data. This time-series behavioral data is categorized by subtask number, standardized, and converted into a model-recognizable instruction sequence using behavioral coding rules, forming an operational sample.

[0010] As a preferred embodiment of the construction robot task planning and skill learning method described in the present invention, the method of constructing operation samples and inputting them into a multimodal task-skill collaborative generation model for skill learning and action instruction generation includes converting each state-action sequence into a condition-action pair represented in text format based on the operation context in the collected and standardized robot time-series operation samples. The text content includes the semantics of the current construction task goal, the summary of the robot's current state, and the corresponding action description under expert instruction, forming an input sequence with a language expression structure. The input sequence is input into the multimodal task-skill collaborative generation model as fine-tuning corpus, and a joint optimization strategy is introduced to simultaneously adjust the parameter coupling structure between the language decoder and the action decoder in the model. The language semantic space and the action control space are synchronously tuned through a bidirectional loss function to construct an action generation head that supports the generalization of multiple types of subtasks. During the inference phase, the action generation head calls the attention mechanism to match the optimal instruction path based on the semantic vector of each subtask's current state, historical execution trajectory, and execution target, and dynamically decides the next control action.

[0011] As a preferred embodiment of the construction robot task planning and skill learning method described in the present invention, the method of fusing multiple sensory data for real-time judgment during construction execution includes continuously acquiring multi-source sensory signals from the robot during construction, performing time synchronization, and fusing them into a unified state vector. A state deviation judgment threshold is set. When the difference between the current state and the model-predicted state exceeds the set threshold, a task state inconsistency flag is triggered, and the current state vector is input into the multimodal task-skill collaborative generation model to generate a new action output.

[0012] As a preferred embodiment of the construction robot task planning and skill learning method described herein, the multimodal task-skill collaborative generation model adjusts instructions based on the latest state and outputs an inference chain, including generating a traceable text description of the inference process while simultaneously generating action instructions based on the current state. This text description records the inference starting point, judgment path, and task decomposition basis according to the decision chain structure, and also includes the results of each subtask assignment and the corresponding triggering logic conditions.

[0013] Another object of the present invention is to provide a construction robot task planning and skill learning system, which can generate instruction modules through learning skills, thereby solving the problems of existing construction robot task execution and control methods, such as the lack of a unified modeling mechanism for natural language and environmental information, the poor generalization ability of skill learning methods that rely on teaching trajectories, and the inability to dynamically feedback and adjust state deviations during task execution, as well as the problem of how to realize the construction robot's intelligent understanding, action generation and real-time reasoning and adjustment of multimodal tasks.

[0014] As a preferred solution of the construction robot task planning and skill learning system described in the present invention, it includes: a target analysis and planning map generation module, a learning skill generation instruction module, and a judgment deviation dynamic adjustment module.

[0015] The target parsing and planning map generation module is used to receive the construction target description and construction environment information through a multimodal task-skill collaborative generation model, generate a structured semantic vector and infer a construction planning map containing multiple subtasks.

[0016] The learning skill generation instruction module is used to collect the operation behavior parameters of the construction robot in each subtask according to the construction planning diagram, construct operation samples and input them into the multimodal task-skill collaborative generation model for skill learning and action instruction generation.

[0017] The judgment deviation dynamic adjustment module is used to integrate multiple perception data to make real-time judgments during the construction execution process. If the task execution status is inconsistent with the planned status, the multimodal task-skill collaborative generation model adjusts the instructions based on the latest status and outputs the reasoning chain.

[0018] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a method for task planning and skill learning of a construction robot.

[0019] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a construction robot task planning and skill learning method.

[0020] Beneficial effects of the present invention: The construction robot task planning and skill learning method provided by the present invention generates a unified structured semantic vector and infers a construction planning diagram by inputting construction task objectives and environmental information into a multimodal task-skill collaborative generation model, and combining natural language semantic parsing, drawing structuring, obstacle information annotation, and robot posture data. This achieves cross-modal fusion of natural language objectives, construction drawing semantics, and environmental perception, and constructs a unified expression form for multi-source heterogeneous data. This achieves the goal of giving construction robots the ability to "understand construction semantics and autonomously plan subtasks," breaking through the traditional limitation of relying solely on structured process template instructions to drive robots, and enabling the system to have stronger task generalization and autonomous planning capabilities.

[0021] By collecting the execution behavior parameters of the construction robot in each subtask, including multi-dimensional data such as position, angle, speed and force, and standardizing and encoding them in time series, they are converted into structured "condition-action pair" text sentences, and input into the large model for fine-tuning training. This has achieved the effect of significantly improving the robot's skill transfer ability and action generation ability in new tasks and complex environments, solving the problems of traditional trajectory learning that cannot be generalized and has a strong dependence on teaching, and reducing the sample requirements and labor costs of model updates.

[0022] By integrating the robot's multi-source perception data such as images, force perception, IMU, and voice during the construction process in real time, the current state vector is constructed and deviations from the expected state are judged. When the state is abnormal, the update mechanism is triggered and the model regenerates instructions and reasoning chains. The system achieves the effect of a self-feedback closed-loop control mechanism with high robustness and low error rate during the construction process, significantly enhancing the flexibility and safety of the task execution process, and breaking through the technical bottleneck of existing technologies that robots cannot adaptively adjust under state fluctuations. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 This is an overall flow chart of a construction robot task planning and skill learning method provided in the first embodiment of the present invention. DETAILED DESCRIPTION

[0025] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0026] Example 1, reference Figure 1 , as one embodiment of the present invention, provides a construction robot task planning and skill learning method, comprising:

[0027] S1: The multimodal task-skill collaborative generation model receives the construction target description and construction environment information, generates a structured semantic vector, and infers a construction plan containing multiple subtasks.

[0028] Construction objectives in natural language are parsed into semantic instruction fragments through an encoding mechanism. Combined with the construction site map, construction drawings, known obstacle information, and the robot's initial position, a unified multidimensional tensor representation is generated through position mapping and semantic fusion. This tensor representation aligns information from different modalities through the cross-modal attention mechanism of the multimodal task-skill collaborative generation model, forming a task-driven semantic vector.

[0029] The multimodal task-skill collaborative generation model retrieves historical task structure templates from the task semantic space based on structured semantic vectors and identifies feasible subtask processes through a graph matching mechanism. Within the identified process, the required tool types, operation sequence, and spatial start and end locations of each subtask are annotated, forming a construction plan consisting of nodes and dependency relationships. Each node represents a subtask, and each edge indicates the sequential execution relationship between subtasks.

[0030] A preferred approach for parsing construction objectives expressed in natural language into semantic instruction fragments through an encoding mechanism involves first performing semantic analysis of the construction objectives described in natural language. Using a large-scale pre-trained model based on the Transformer architecture, the semantic components of the task, such as key verbs, target objects, and action sequences, are extracted and converted into intermediate representations of the task instruction fragments. Simultaneously, the construction site map and construction drawing data are vectorized and aligned with obstacle information using a spatial registration method to obtain the environment topology. These semantic instructions and environmental information are then fused using a position mapping mechanism to form a unified multidimensional tensor representation describing the "target-site-path" triple relationship within the current task scenario. This multidimensional tensor is then fed into a multimodal task-skill co-generation model. Internally, a cross-modal attention mechanism aligns the language semantics with the environmental structure, enabling the model to understand the relevance of various inputs within a shared semantic space. This process enables the large model to obtain a task-driven semantic vector representing the optimal task representation given the current construction objective and environmental conditions.

[0031] A preferred approach for a multimodal task-skill collaborative generation model to retrieve historical task structure templates from the task semantic space based on structured semantic vectors specifically includes: Based on the semantic vectors, the multimodal task-skill collaborative generation model enters the task reasoning phase. The model retrieves historical task structure templates (e.g., standard operating procedures for wall masonry, rebar tying, etc.) from the task semantic space and compares the structural relationships between the current task and known templates using semantic similarity and graph structure matching algorithms to identify adaptable task processes. After a successful match, the model automatically decomposes the task, dividing the overall construction goal into several subtasks and annotating key execution parameters for each subtask, such as the required execution tool type, operation sequence, and spatial start and end locations. Ultimately, all identified and parsed subtasks and their execution dependencies are organized into a construction plan. The plan is represented as a graph structure, with nodes representing individual subtask units and edges representing the dependency order and triggering logic between tasks. This plan serves as the basic input for subsequent robot skill invocation and execution scheduling, ensuring that construction robots can efficiently complete task decomposition and instruction scheduling in complex scenarios.

[0032] Furthermore, a preferred solution for a multimodal task-skill collaborative generation model specifically includes three components: a semantic encoder, an environment fusion encoder, and a task graph decoder. These three components are linked through a cross-modal attention mechanism to collaboratively achieve semantic understanding and task structure generation.

[0033] The semantic encoder performs in-depth semantic analysis on the natural language input of the construction objectives. This component employs a multi-layered, serialized language encoding unit structure. Each layer incorporates word embedding, context-dependent modeling, and an attention mechanism to extract key semantic components from the text, such as verbs, objects, and operational instructions. Ultimately, a unified semantic vector is formed, serving as the semantic representation of the task objective.

[0034] The environment fusion encoder is primarily responsible for uniformly encoding construction drawings, obstacle information, and the robot's initial pose. First, the drawing structure is converted into a structural vector using graph structure embedding technology. Obstacle information, in the form of a spatial point cloud, is input into a sparse feature extraction network for spatial compression. The initial pose is directly mapped into a six-dimensional spatial representation. After these multi-source data pass through the dimensional alignment and fusion module, a fused vector representing the current state of the construction environment is output.

[0035] The task graph decoder receives the outputs from the semantic encoder and the environment fusion encoder. It internally employs a cross-modal attention mechanism to interactively fuse semantic and spatial information, and combines historical task templates for structural reasoning. The model first performs weighted matching of semantic and environment vectors. It then uses a template retrieval mechanism to locate the closest historical task structure and dynamically adjusts subtask nodes and execution relationships based on the current semantic intent. The final output is a task planning graph, in which each node represents an executable subtask, and each edge represents the dependency order and logical connections between subtasks. This structural graph provides clear task boundaries and contextual constraints for subsequent action generation and skill learning.

[0036] The various components of the aforementioned model structure form a closed-loop interaction. The semantic encoder provides semantic goals, the environmental encoder provides spatial conditions, and the decoder fuses these two to generate an actionable task structure diagram. The model possesses the ability to jointly understand task language and complex construction environments, and supports flexible structure matching and dynamic task decomposition, ensuring that the robot can generate a task path diagram with execution logic based on multimodal input.

[0037] Furthermore, the model structure design fully considers the heterogeneity and modality of the input data. While traditional neural network models are often constructed for a single modality, the semantic encoder, environment fusion encoder, and trajectory behavior module employed in this invention are specifically modeled for different types of data, such as natural language, graph structures, spatial point sets, and poses. After each channel is independently encoded, a cross-modal attention mechanism is used to learn representations in a shared semantic space, ensuring that heterogeneous data can be fused and fed into a unified decoder for task graph prediction or action generation.

[0038] It should be noted that S1 uses semantic parsing and multimodal fusion mechanisms to uniformly encode natural language task objectives and environmental information, construct task-driven semantic vectors and generate construction planning drawings, realizing automatic conversion from human language to robot-executable tasks, improving task understanding accuracy and planning intelligence, and ensuring that the robot can efficiently adapt to complex scenarios.

[0039] S2: Collect the operation behavior parameters of the construction robot in each subtask according to the construction plan, construct operation samples and input them into the multimodal task-skill collaborative generation model for skill learning and action command generation.

[0040] While the robot is executing a specified subtask, the end effector position, joint angle, velocity vector and real-time force feedback information are recorded at fixed time intervals, and status annotations are performed at key operation points to form time-series behavior data.

[0041] The behavior collection tensor definition is expressed as:

[0042] ;

[0043] Among them, the behavior tensor Indicates the number The robot in the subtask number The complete behavior flow in and Indicates the task execution time period. Represents a three-dimensional velocity vector. is the joint angle gradient change tensor. is the position state function of the end effector. is the force feedback intensity collected by the sensor, is the unit impedance function, which represents the actuator's current ability to respond to external disturbances.

[0044] Expressed as:

[0045] ;

[0046] in, represents the initial displacement. Indicates the current speed of the robot.

[0047] Expressed as:

[0048] ;

[0049] in, Indicates The force feedback intensity collected by the sensor at all times. Represents a robot The impedance adjustment coefficient.

[0050] The time series behavior data is classified according to the subtask number and then standardized, which is expressed as:

[0051] ;

[0052] in For the number Standardized behavioral sample sequence, Indicates the sample The rate of change of joint units, Indicates the total number of sample frames.

[0053] The behavioral coding rules are converted into instruction sequences that can be recognized by the model to form operation samples. In the collected and standardized robot time-series operation samples, each state-action sequence is converted into a condition-action pair expressed in text format based on the operation context. The text content includes the semantics of the current construction task goal, the summary of the current robot state, and the corresponding action description under expert instruction, forming an input sequence with a language expression structure, which is expressed as:

[0054] ;

[0055] in, For the final generated textual sentence structure, Represents the formatting function, is the semantic vector of the construction target, is the Sigmoid filter value of the standardized behavior sequence, It is the structural description vector of the current action under expert teaching.

[0056] The input sequence serves as fine-tuning corpus for the multimodal task-skill collaborative generation model. Furthermore, the training process for the multimodal task-skill collaborative generation model involves first inputting the natural language description of the construction task into a semantic encoder. During training, the encoder extracts the semantic instructions, target entities, and operational behaviors from the sentence layer by layer through multiple rounds of contextual association modeling, ultimately outputting a unified semantic embedding vector. This vector serves as the fundamental representation for the model's understanding of the task objectives during training.

[0057] Subsequently, the construction drawing information, the obstacle point set, and the initial robot pose serve as environmental inputs and are processed by the environment fusion encoder. The drawing structure is vectorized using a graph neural network, while obstacle information is extracted using a sparse point processing network for spatial feature extraction. The robot pose is directly mapped to a spatial state representation. These three elements are integrated into a unified environmental representation through a fusion mechanism to provide spatial constraints.

[0058] The semantic vector and the environment vector are simultaneously input into the task graph decoder. During training, the decoder uses its internal attention matching mechanism and combines it with the actual task structure labels to adjust the generated results. By comparing the generated results with manually annotated task structure graphs, the system optimizes the accuracy of node generation and the rationality of the task flow layer by layer. The training goal is to enable the model to accurately reproduce the subtask breakdown structure and execution order of real construction tasks.

[0059] Model training is optimized using a staged structural loss function, independently supervising the classification accuracy of task nodes and the structural connectivity between subtasks. The node recognition component assesses whether the model correctly identifies the type and order of each subtask, while the structural relationship component determines the topological consistency between the task dependency graph generated by the model and the manual task graph. Furthermore, to enhance the robustness of the model, a regularization term is added to limit overfitting.

[0060] The entire training process utilizes a batch gradient optimization strategy. During each training round, an error feedback mechanism is used to adjust the internal parameters of the semantic encoding layer, the environment encoding layer, and the task graph decoding layer. This allows for the coordinated optimization of multimodal information fusion and structured task generation. The trained model is able to automatically output a well-structured and logically coherent construction subtask flowchart based on natural language objectives and environmental conditions, given new task inputs.

[0061] A joint optimization strategy is introduced to simultaneously adjust the parameter coupling structure between the language decoder and the action decoder in the model. A bidirectional loss function is used to simultaneously optimize the language semantic space and the action control space, building an action generation head that supports generalization across multiple subtask types. During the inference phase, the action generation head uses the attention mechanism to match the optimal instruction path based on the semantic vector of each subtask's current state, historical execution trajectory, and execution target, and dynamically decides on the next control action. The action generation vector inference is represented as:

[0062] ;

[0063] in Indicates time Always focus on subtasks Output action control vector. Represents the historical trajectory backtracking function. is the target guidance function. Represents the function for the current state. is the action encoding function.

[0064] Expressed as:

[0065] ;

[0066] in, Indicates the length of the historical trajectory window. For robots in time The trajectory state vector.

[0067] Expressed as:

[0068] ;

[0069] in, Represents the target position vector. Represents the robot's current estimated position vector. Represents the target error adjustment coefficient.

[0070] Expressed as:

[0071] ;

[0072] in, Represents the dimension of the robot's current state vector. Indicates the robot Dimension the current state variable value.

[0073] A preferred solution of the bidirectional loss function specifically includes: the language-action joint optimization loss function is expressed as:

[0074] ;

[0075] in, is the joint loss function. is the number of sample batches. and are the original and predicted semantic decoding vectors, respectively. and is the action space output and prediction distribution. represents the KL divergence metric, and is the sample feature complexity adjustment factor.

[0076] It should be noted that S2 accurately captures the dynamic execution characteristics of the construction robot in each subtask by constructing a behavior acquisition tensor, and introduces standardization and textualization mechanisms to convert low-level control signals into language expression structures that can be understood by multimodal generative models, thereby establishing a "state-semantics-action" triple alignment mechanism. Compared with existing learning methods that rely on pure teaching trajectories or vector instructions, this step realizes cross-modal transfer training and language space generalization, and adjusts the parameter coupling of language and action space through a joint optimization mechanism, significantly improving the action instruction generation capability and response speed under new tasks. This method breaks through the technical bottlenecks of traditional robot models' poor migration capabilities and high teaching costs in new scenarios, and has stronger scalability and intelligent adaptability.

[0077] S3: During the construction execution process, multiple sensory data are integrated to make real-time judgments. If the task execution status is inconsistent with the planned status, the multimodal task-skill collaborative generation model adjusts the instructions based on the latest status and outputs the reasoning chain.

[0078] During the construction process, the robot continuously acquires multi-source perception signals, synchronizes them in time, and fuses them into a unified state vector. A state deviation threshold is set. When the difference between the current state and the model's predicted state exceeds the set threshold, a task state inconsistency flag is triggered, and the current state vector is input into the multimodal task-skill collaborative generation model to generate a new action output.

[0079] Furthermore, as the construction robot performs its construction tasks, the system continuously collects, fuses, and assesses the state of multiple sensory signals in real time to achieve dynamic task execution control and model feedback updates. Sensory data includes, but is not limited to, visual image streams, force sensor output, inertial measurement unit (IMU) data, depth camera point cloud information, and voice interaction input. All data is aligned using a timestamp synchronization mechanism to construct a unified time-series state vector.

[0080] This state vector undergoes state fusion at a fixed period (every 50ms), undergoing modal alignment and feature compression to form a current state representation vector of uniform dimension. At the same time, the system retains the expected state vector inferred from the task planning graph by the multimodal task-skill collaborative generation model.

[0081] To determine whether the robot's current execution state deviates from the expected planned path, the system sets the state deviation judgment threshold to 0.12, which is determined based on the empirical optimal solution during the model tuning process. The system calculates the Euclidean distance between the current state and the predicted state in each cycle:

[0082] ;

[0083] in, represents the current state representation vector of uniform dimension, represents the expected state vector.

[0084] like If the error is greater than 0.12, the task execution status is considered inconsistent with the planned status, triggering a state update process. This process involves freezing the current execution thread and saving live data. The latest state vector is then fed into the model, and action instructions are regenerated based on the historical trajectory and target semantics. The model then outputs a reasoning chain containing the basis for the adjustment, and the plan map is updated.

[0085] at this time It will be passed back to the multimodal task-skill collaborative generation model as the latest state input, triggering the immediate reasoning process.

[0086] In this process, the model combines the current task semantics, historical execution records, and subtask context states to regenerate the next action instruction and simultaneously generate a traceable reasoning chain text description. The reasoning chain is represented by a tree structure and contains the following elements:

[0087] Reasoning starting point: current state input summary.

[0088] Judgment path: The semantic backtracking and strategy matching path performed within the model (e.g., “the current state deviates from the target trajectory → detects abnormal load on the actuator arm → adjusts the path”).

[0089] Task decomposition basis: Indicates how the model re-decomposes the task based on the template in the task library (such as "sub-task structure based on obstacle avoidance").

[0090] Subtask allocation results: the updated execution responsibility allocation and estimated time for each task node.

[0091] Trigger logic conditions: Mark the status as "active reconstruction" or "passive repair", and record the trigger threshold and execution timestamp.

[0092] While the multimodal task-skill collaborative generation model generates action instructions based on the current state, it also generates a traceable text description of the reasoning process. This text description records the reasoning starting point, judgment path, and task breakdown basis according to the decision chain structure, and also includes the results of each subtask assignment and the corresponding triggering logic conditions.

[0093] Furthermore, the output of the inference chain is synchronously sent to the control terminal and the supervision system, and is automatically archived to the model iteration training pool as incremental data samples for subsequent fine-tuning and reinforcement learning stages.

[0094] In the present invention, the image state deviation trigger threshold is set to 0.08, the force feedback abnormality threshold is set to 3.5N, the end position drift threshold is set to 5mm, and the IMU angular velocity change rate threshold is set to 0.5rad / s.

[0095] It should be noted that S3 dynamically adjusts the robot's execution strategy by integrating multi-source perception data in real time and comparing it with the expected task plan status. When deviations are detected, an inference chain is immediately generated and action instructions are optimized, ensuring the robot's flexible adaptation in complex environments. The traceability of the inference chain improves the interpretability and fault tolerance of task execution, ensuring efficient and accurate construction control, and overcoming the poor adaptability of traditional methods in dynamic environments.

[0096] Example 2 is an embodiment of the present invention, which provides a construction robot task planning and skill learning system, including a target analysis and planning map generation module, a skill learning and instruction generation module, and a judgment deviation dynamic adjustment module.

[0097] The target parsing and planning map generation module is used to receive the construction target description and construction environment information through the multimodal task-skill collaborative generation model, generate a structured semantic vector and infer a construction planning map containing multiple subtasks.

[0098] The module for parsing the target and generating the planning map includes a semantic parsing submodule and a task planning map generating submodule.

[0099] Furthermore, the semantic parsing submodule receives construction task objectives in natural language, encodes the target semantics into semantic vectors using a Transformer architecture model, and generates a unified input tensor based on information such as construction drawings, site topology, and the robot's initial pose. The task planning graph generation submodule infers a construction plan containing multiple subtasks based on the input tensor. The plan represents the execution dependencies between nodes in the form of a graph structure, with each node corresponding to an atomic construction operation, and edges representing the task sequence and logical conditions.

[0100] It should be noted that the semantic parsing submodule is the starting point for the target parsing and plan generation module. The semantic representation it outputs serves as the input for subsequent subtask reasoning. The task plan generation submodule, based on historical task templates and a structured matching mechanism, completes the structural representation of the current construction task, providing a logically clear task framework for the skill learning phase.

[0101] It should also be noted that the module for parsing goals and generating planning diagrams serves as the information entry of this system, completing the cross-modal conversion from natural language goals to structured task diagrams, and is the basis of the intelligent planning capability of the present invention.

[0102] The learning skill generation instruction module is used to collect the operation behavior parameters of the construction robot in each subtask according to the construction planning diagram, construct operation samples and input them into the multimodal task-skill collaborative generation model for skill learning and action instruction generation.

[0103] The learning skill generation instruction module includes a behavior collection and encoding submodule and an action instruction generation submodule.

[0104] Furthermore, the behavior acquisition and encoding submodule collects behavioral parameters such as position, angle, and force feedback during the robot's subtask execution, normalizes and time-aligns these parameters, and then encodes them into standardized "condition-action pairs" based on the task context structure. The action command generation submodule feeds standardized behavior samples into the multimodal task-skill collaborative generation model. Through a semantic-action space collaborative optimization mechanism, it trains and fine-tunes the model to generate action command outputs suitable for various subtasks.

[0105] It should be noted that the behavior acquisition and encoding submodule not only ensures the structural consistency of the original action sample data but also provides semantically relevant training data for model fine-tuning. The action instruction generation submodule, as the core module for skill transfer, supports the generation of generalized execution strategies for untaught tasks.

[0106] It should also be noted that the learning skill generation instruction module plays the role of "from experience to strategy" in the present invention, enabling the robot to have continuous learning and dynamic adaptability when facing different construction tasks, and is a key link in realizing intelligent skill transfer.

[0107] The judgment deviation dynamic adjustment module is used to integrate multiple perception data for real-time judgment during the construction execution process. If the task execution status is inconsistent with the planned status, the multimodal task-skill collaborative generation model adjusts the instructions based on the latest status and outputs the reasoning chain.

[0108] The judgment deviation dynamic adjustment module includes a state fusion judgment submodule and an inference chain generation submodule.

[0109] Furthermore, the state fusion judgment submodule is used to collect multimodal perception data (such as visual images, IMU posture, and force sensor information) in real time during the construction process, perform time synchronization and feature fusion, construct a unified state vector, and compare it with the expected state. If the deviation exceeds a set threshold, a state inconsistency flag is triggered. The inference chain generation submodule is used to re-input the current state into the multimodal generation model when state deviation occurs, generate new action instructions, and simultaneously output a traceable inference chain text. The inference chain records the task reconstruction path, judgment basis, and update strategy.

[0110] It should be noted that the state fusion judgment submodule ensures the robot's real-time perception and deviation recognition capabilities for sudden changes in complex construction environments. The inference chain generation submodule enhances the interpretability of the model's decision-making process and provides an interface for human intervention and a source of incremental training samples.

[0111] It should also be noted that the judgment deviation dynamic adjustment module realizes the "state deviation-action reconstruction-logic traceability" closed-loop feedback mechanism, which is the guarantee mechanism of the present invention to support the execution of high-robustness tasks, and significantly improves the system's environmental adaptability and execution flexibility.

Claims

1. A construction robot task planning and skill learning method, characterized in that: include: The multimodal task-skill collaborative generation model receives the construction target description and construction environment information, generates a structured semantic vector, and infers a construction plan diagram containing multiple subtasks. According to the construction plan, the construction robot's operational behavior parameters in each subtask are collected, and operation samples are constructed and input into the multimodal task-skill collaborative generation model for skill learning and action command generation. During the construction execution process, multiple sensory data are integrated to make real-time judgments. If the task execution status is inconsistent with the planned status, the multimodal task-skill collaborative generation model adjusts the instructions based on the latest status and outputs the reasoning chain; Constructing operation samples and inputting them into a multimodal task-skill collaborative generation model for skill learning and action instruction generation involves expressing the operation samples into condition-action pairs through textual conversion, inputting them into the multimodal task-skill collaborative generation model for training, and employing a joint optimization strategy to simultaneously fine-tune parameters in both the language semantic space and the action control space to construct an action generation head with skill generalization capabilities. When the model generates each action instruction, it dynamically determines the next action to be executed based on the current subtask status, historical operation trajectory, and execution goal through an attention mechanism. Generating structured semantic vectors includes, The construction objectives in natural language are parsed into semantic instruction fragments through an encoding mechanism. Combined with the construction site map, construction drawings, known obstacle information and the robot's initial position, a unified multi-dimensional tensor representation is generated through position mapping and semantic fusion. The multi-dimensional tensor representation aligns information from different modalities through the cross-modal attention mechanism of the multimodal task-skill collaborative generation model to form a task-driven semantic vector.

2. The construction robot task planning and skill learning method according to claim 1, wherein: The inference of a construction planning diagram containing multiple subtasks includes: The multimodal task-skill collaborative generation model retrieves historical task structure templates in the task semantic space based on structured semantic vectors and identifies feasible subtask processes through a graph matching mechanism. In the determined process, the types of tools required for each subtask, the operation sequence, and the spatial start and end positions are marked to form a construction planning diagram consisting of nodes and dependency relationships. Each node represents a subtask, and each edge represents the execution sequence relationship between subtasks.

3. The construction robot task planning and skill learning method according to claim 2, characterized in that: The operation behavior parameters of the construction robot in each subtask collected according to the construction plan include: While the robot is performing a specified subtask, it records the end effector position, joint angle, velocity vector, and real-time force feedback information at fixed time intervals, and annotates the status at key operation points to form time-series behavior data. The temporal behavior data is classified according to the subtask number and then standardized, and converted into an instruction sequence that can be recognized by the model through behavior coding rules to form an operation sample.

4. The construction robot task planning and skill learning method according to claim 3, wherein: The construction of operation samples and inputting them into the multimodal task-skill collaborative generation model for skill learning and action instruction generation includes: In the collected and standardized robot time-series operation samples, each state-action sequence is converted into a condition-action pair expressed in text format based on the operation context. The text content includes the semantics of the current construction task goal, a summary of the robot's current state, and a description of the corresponding action under expert instruction, forming an input sequence with a language expression structure. The input sequence is fed into a multimodal task-skill collaborative generation model as fine-tuning corpus. A joint optimization strategy is introduced to simultaneously adjust the parameter coupling structure between the language decoder and the action decoder in the model. A bidirectional loss function is used to simultaneously optimize the language semantic space and the action control space, constructing an action generation head that supports generalization across multiple subtasks. During the inference phase, the action generation head calls the attention mechanism to match the optimal instruction path based on the current state of each subtask, the historical execution trajectory, and the semantic vector of the execution target, and dynamically decides the next control action.

5. The construction robot task planning and skill learning method according to claim 4, characterized in that: The integration of multiple sensory data for real-time judgment during the construction execution process includes: During the construction process, the robot continuously acquires multi-source perception signals, synchronizes them in time, and fuses them into a unified state vector; Set the state deviation judgment threshold. When the difference between the current state and the model predicted state exceeds the set threshold, the task state inconsistency flag is triggered, and the current state vector is input into the multimodal task-skill collaborative generation model to generate a new action output.

6. The construction robot task planning and skill learning method according to claim 5, characterized in that: The multimodal task-skill collaborative generation model adjusts instructions based on the latest state and outputs the reasoning chain, including: While the multimodal task-skill collaborative generation model generates action instructions based on the current state, it also generates a traceable text description containing the reasoning process; The text description records the reasoning starting point, judgment path and task decomposition basis according to the decision chain structure, and is accompanied by the results of each subtask allocation and the corresponding triggering logic conditions.

7. A construction robot task planning and skill learning system using the method according to any one of claims 1 to 6, characterized in that: It includes a module for analyzing goals and generating a planning map, a module for learning skills and generating instructions, and a module for dynamically adjusting judgment deviations; The target parsing and planning diagram generation module is used to receive the construction target description and construction environment information through the multimodal task-skill collaborative generation model, generate a structured semantic vector and infer a construction planning diagram containing multiple subtasks; The learning skill generation instruction module is used to collect the operation behavior parameters of the construction robot in each subtask according to the construction plan, construct operation samples and input them into the multimodal task-skill collaborative generation model for skill learning and action instruction generation; The judgment deviation dynamic adjustment module is used to integrate multiple perception data to make real-time judgments during the construction execution process. If the task execution status is inconsistent with the planned status, the multimodal task-skill collaborative generation model adjusts the instructions based on the latest status and outputs the reasoning chain.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the construction robot task planning and skill learning method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the construction robot task planning and skill learning method described in any one of claims 1 to 6 are implemented.

Images