Robot control method, system and equipment based on multi-modal large model and medium

Abstract

The invention relates to the technical field of robot control, and discloses a robot control method, system, equipment and medium based on a multi-modal large model, and the method comprises the steps: collecting the multi-source modal data of a scene where an operation task is located, and carrying out the processing through a machine learning model, obtaining a multi-modal feature, and carrying out the position coding and Transform fusion processing, multi-modal fusion features are obtained, the multi-modal fusion features and the constructed job task knowledge base are input into a large language model to decompose a target job task, a human-in-the-loop mechanism is introduced to optimize a decomposition result, and a sub-task sequence is obtained; according to a subtask type in the subtask sequence, processing the subtask sequence through a visual language action model or a reinforcement learning model, and generating a motion instruction to enable the robot to start an execution process of the target operation task; live-line work tasks are processed through the multi-modal large models LLM, VLA and the like, and the work efficiency of the autonomous distribution network live-line work robot is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, in particular to a robot control method and system based on a multi-modal large model, a device and a medium. BACKGROUND

[0002] With the continuous development of technology, robots are often applied to outdoor live working scenes with high risk factors such as harsh weather conditions, high altitude, and strong voltage fields. However, the existing autonomous distribution live working robots still need manual cooperation of operators to adjust the position of the robot. After confirming that the position is appropriate through the video screen of the mobile control terminal, the specific step program of the corresponding working project is started, and the manual intervention is highly dependent, which further leads to low working efficiency of the robot.

[0003] Therefore, how to solve the problem of low working efficiency of the existing autonomous distribution live working robot has become a technical problem to be solved by those skilled in the art. SUMMARY

[0004] The present application provides a robot control method and system based on a multi-modal large model, which solves the problem of how to improve the working efficiency of the existing autonomous distribution live working robot.

[0005] To solve the above technical problems, the present application provides a robot control method based on a multi-modal large model, comprising: Real-time acquisition of multi-source modal data of a scene where a target working task is located, and based on the type of the multi-source modal data, a corresponding machine learning model is used to extract features from the multi-source modal data to obtain multi-modal features; Position encoding and Transformer fusion processing are performed on the multi-modal features to obtain multi-modal fusion features; The constructed working task knowledge base and the multi-modal fusion features are input into a large language model to obtain a decomposition result of the target working task, and a human-in-the-loop mechanism is introduced to optimize the decomposition result to obtain a subtask sequence; Based on the subtask type in the subtask sequence, the subtask sequence and the multi-source modal data are input into a visual language action model or a reinforcement learning model for processing to generate motion instructions corresponding to the subtask sequence, so that the robot starts the execution process of the target working task.

[0006] Compared with the prior art, the beneficial effects of the present application are as follows: The feature extraction can be performed on different types of data by corresponding machine learning models, which can more comprehensively and accurately capture scene information, compared to single modal data processing, and provides a richer feature basis for subsequent task decomposition and instruction generation, and improves the understanding ability of complex work scene; the multi-modal fusion and task decomposition are combined, and the human-in-the-loop mechanism is introduced to optimize the decomposition result, which fully utilizes the language understanding and reasoning ability of the large language model, so that the task decomposition is more in line with the actual demand and logic, and the accuracy and rationality of the task decomposition are improved; the dynamic model selection mechanism can play the advantages of different models according to the characteristics of different sub-tasks, improve the efficiency and accuracy of instruction generation, better control the robot to execute complex work tasks, and further improve the work efficiency of the existing autonomous power distribution live work robot. BRIEF DESCRIPTION OF DRAWINGS

[0007] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0008] Figure 1 is a flowchart of a robot control method based on a multi-modal large model provided by an embodiment of the present application; Figure 2 is a structural diagram of a robot control system based on a multi-modal large model provided by an embodiment of the present application; Figure 3 is a structural diagram of an electronic device provided by an embodiment of the present application; Reference signs: Among them, 10, feature extraction module; 20, feature fusion module; 30, task decomposition module; 40, instruction execution module; 5000, electronic device; 5001, processor; 5002, bus; 5003, memory; 5004, transceiver. DETAILED DESCRIPTION

[0009] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings and embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0010] In the description of the present application, the terms "first", "second", "third" and the like are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second", "third" and the like can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise stated, the meaning of "multiple" is two or more. The term "and / or" used herein includes any and all combinations of one or more related listed items. The specific meaning of the above terms in the present application can be understood according to the specific circumstances by those skilled in the art.

[0011] In the description of the present application, it should be noted that, unless otherwise defined, all technical and scientific terms used in the present application have the same meaning as generally understood by those skilled in the art. The terms used in the description of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The specific meaning of the above terms in the present application can be understood according to the specific circumstances by those skilled in the art.

[0012] The power line live working robot refers to a robot system for carrying out power line live working by human-machine cooperation or self-service, which is composed of a robot body, a working tool and an insulating bearing platform. Among them, under the assistance of ground operators, the power line live working robot can realize identification and positioning, path planning, end tool replacement, and automatically execute live working tasks, which is called autonomous power line live working robot, hereinafter referred to as "robot".

[0013] At present, the robot needs to complete the live working project in the typical working scene. The typical scene is single return horizontal arrangement line, single return triangular arrangement line and double return vertical arrangement line. The working projects that can be carried out are breaking and connecting the current-carrying wire, installing grounding hanging ring, installing fault indicator, pruning branches, supporting insulator, live assisting installation or removal of insulating shield, live replacing lightning arrester, live replacing fuse and live replacing straight pole insulator. However, the current robot has high operation complexity and relies on manual intervention. In most cases, the robot needs to be manually adjusted by the operator, and the position and working point are confirmed by the mobile control terminal. The robot cannot completely complete the task autonomously. With the increase of the joint freedom degree of the robot and the diversification of the task, the traditional planning method based on geometry or simplified dynamics model adopted by the robot cannot find a feasible solution in high-dimensional and multi-constrained scenes, and cannot realize human-like dexterous operation, which leads to insufficient dexterity of the robot and further low working efficiency.

[0014] Based on this, in an embodiment, as Figure 1As shown, the first aspect of the present application provides a robot control method based on a multi-modal large model, comprising: S1, real-time acquisition of multi-source modal data of the scene where the target task is located, and based on the type of the multi-source modal data, a corresponding machine learning model is used to extract features from the multi-source modal data to obtain multi-modal features; wherein the multi-source modal data includes force sensation and point cloud data, image data and audio data; At present, the actuator of the robot is a pair of mechanical arms carrying dexterous hands, and the robot also carries multiple types of sensors, including visual sensors, force sensors, tactile sensors, microphones, laser radars and environmental sensors (humidity sensors, wind speed and direction sensors, etc.), to real-time collect multi-source modal data of the scene where the target task is located, including force sensation and point cloud data, image data, audio data, distance data, real-time sensor data, environmental data, and robot joint torque, speed, temperature and robot end position and attitude.

[0015] For the collected raw data, data preprocessing techniques are used, such as filtering and denoising of images, to ensure the high quality and usability of the data; through timestamp synchronization technology, the consistency of multi-modal data such as laser radar, millimeter wave radar, visual sensor, force sensor, etc. on the time axis is ensured, and the data mismatch problem caused by time difference is reduced; and Kalman filter algorithm is used to perform real-time data fusion and correction on the raw data of laser radar, millimeter wave radar and visual sensor, to improve the accuracy and robustness of the data fusion between different sensors.

[0016] In an embodiment, the use of a corresponding machine learning model to extract features from the multi-source modal data to obtain multi-modal features comprises: inputting the force sensation and point cloud data into a long short-term memory network model for processing to capture force interaction patterns and obtain time series dynamic features; inputting the image data into an improved ResNet-50 model for processing to obtain local features of the target object; inputting the audio data into a Whisper-Base model and a BERT-Base-Chinese model for processing to obtain speech features for parsing execution instructions; combining the time series dynamic features, the local features and the speech features to obtain the multi-modal features.

[0017] Specifically, for the collected multi-source modal data of vision, force sensation and environment, based on the type thereof, multiple machine learning models are used to extract features from the multi-modal data, and the specific process comprises: The force sensor and the force sensation and point cloud timing data collected by the laser radar are input into a long short-term memory network (LSTM) for processing to extract dynamic changes and correlations in the time dimension, capture the force interaction mode, the LSTM gate mechanism retains long-term dependent information and forgets irrelevant information, generates time sequence features with generalization ability, helps the robot understand the force interaction mode and the change of the three-dimensional structure of the environment, and outputs the time sequence dynamic features. The processing process of the LSTM on the time sequence data can refer to the application of the LSTM in the prior art, and will not be described in detail here.

[0018] The RGB image data collected by the visual camera is input into an improved ResNet-50 model for processing, and local features containing the edges, textures, colors and geometric features of the target object (such as a wire and a clamp) are output; wherein the improved ResNet-0 model optimizes the convolutional layer and the residual block on the basis of the original ResNet-50, and enhances the model's ability to extract target edge features. It takes ResNet-50 as a feature extraction backbone network, which takes a 640x480 pixel RGB image as input, normalizes it (pixel scaling to [0, 1]) and data augmentation (random cropping, flipping, brightness adjustment) preprocessing, and then extracts low-level features in the image by a 7x7 convolution kernel (stride 2) in the model, and then extracts high-level semantic features by four residual modules (3, 4, 6, 3 residual blocks) step by step, uses residual connection to alleviate gradient disappearance, and finally compresses high-dimensional features into a 512-dimensional vector by global average pooling to retain the spatial position and context information of the target, ensuring accurate recognition and positioning in strong light, shadow or occlusion scenes.

[0019] The audio data of 16kHz, 16-bit PCM audio collected by the microphone is separated from the speech and background noise by WebRTC VAD (based on short-time energy and zero-crossing rate), the obtained speech segments are input into the Whisper-Base model (74M parameters, Chinese optimization), combined with Mel spectrum features and beam search (beam width 5) to convert to high-precision text, then the obtained text is segmented and encoded by BERT-Base-Chinese (12-layer Transformer, 768-dimensional hidden layer) to generate a 768-dimensional semantic vector, capturing deep semantic instructions (such as grabbing a clamp and installing it to a wire), mapping to the robot task space, ensuring robust human-computer interaction, and outputting speech features for parsing and executing instructions.

[0020] Finally, the multi-modal features can be obtained by combining the time series dynamic features, local features and speech features extracted by the machine learning model. It should be noted that the above feature extraction process can also be performed using other models, such as using the AlexNet model to extract image features. Of course, speech features and time series dynamic features can also be extracted using other models. The extraction process refers to the implementation process of the corresponding model in the prior art, and will not be described here.

[0021] The present application adopts machine learning models with strong adaptability and strong pertinence to process different types of multi-source modal data. This model selection method according to data characteristics fully utilizes the expertise of each model and improves the quality and efficiency of feature extraction. By simultaneously extracting the time series dynamic features of force and point cloud, the geometric features of image and the speech features of audio and video, comprehensive and rich information is provided for subsequent task processing. The improved classic model (such as improved ResNet-50) and the combination of excellent models in different fields (such as Whisper-Base in the speech field and BERT-Base-Chinese in the text field) applied to multi-modal feature extraction reflect innovative technical ideas and can better adapt to specific task requirements. Model combination fully utilizes the advantages of different models, realizes functional complementation and provides a new effective way to solve complex multi-modal data processing problems.

[0022] S2, position encoding and Transformer fusion processing are performed on the multi-modal features to obtain multi-modal fusion features; In an embodiment, step S2 comprises: The time steps are embedded in the multi-modal features by position encoding to obtain multi-modal encoding features; The multi-modal encoding features are input into a Transformer encoder to perform feature fusion through a self-attention mechanism and a multi-head attention mechanism to obtain the multi-modal fusion features; Wherein, the position encoding is performed by the following formula: t ′= t + t In the formula, is the position encoding vector of the tth time step; d is the feature vector dimension of each time step in the multi-modal features; t ′ is the multi-modal encoding feature; t is the multi-modal feature.

[0023] Specifically, the present application adds position information to each time step of the multi-modal feature by position encoding, so that the Transformer can distinguish the sequence order: the multi-modal features such as visual features, force perception features and language features are combined into a sequence, and the length and dimension of the sequence are determined, an integer index t is assigned to each position of the multi-modal feature, for each position t, a d-dimensional position encoding vector is calculated, for even index (0, 2, 4, , d-2), calculate ; for odd index (1, 3, 5, , d-1), calculate ; finally, the position encoding vector t is added to the feature vector t at the corresponding position, that is, the multi-modal encoding feature t ′.

[0024] In the process of Transformer feature fusion, the present application introduces position encoding (Positional Encoding) to provide sequence position information to assist the model in understanding time series and sequence order. Since multi-modal data (especially force / point cloud, etc.) has time series, the time step information is embedded into the feature vector through additive position encoding, and the Transformer can identify the input at different time points or spatial order, so as to consider the time series context during fusion. For example, for multi-frame point clouds in the process of executing an action by a mechanical arm, the position encoding can help the Transformer to distinguish early and late frames, so that the fused representation retains the trajectory of the gradual change of the environment.

[0025] Position encoding is very important in feature fusion, because it can preserve the temporal information. Point cloud and force data change over time (e.g. point cloud sequence when the robot is grasping). Position encoding adds a unique time identifier to each frame, and the Transformer distinguishes between early and late frames, preserving dynamic trajectories (e.g. wire clip movement path), while position encoding can distinguish the order of modalities. Visual, force, and language features are arranged in a fixed order (e.g. visual 0-20, force 21-40, language 41-63), ensuring that the Transformer identifies the modality source and correctly associates the instruction "install" with the wire position in the vision. This encoding can significantly enhance context modeling, allowing the Transformer to capture temporal context, such as the association of early visual features (wire clip position) with late force features (grasping feedback), improving the semantic integrity of the fused representation. Ultimately, this can improve the robustness of feature fusion. In contrast to many schemes that ignore timing, this scheme can explicitly model time steps, adapting to scenarios where action sequences are strictly ordered (e.g. grasping before installation), ensuring temporal consistency in the fused representation.

[0026] To fully utilize the information of each modality and obtain unified environmental representation, the scheme adopts a Transformer network structure as the core architecture of multi-modal feature fusion. The Transformer encoder can efficiently model the complex interdependence between different modalities, and process the input multi-modal encoding features to fuse the input features through self-attention and multi-head attention mechanisms to obtain multi-modal fusion features.

[0027] The self-attention mechanism determines the importance of each position's features to other position features by calculating the similarity between query vectors (Query), key vectors (Key), and value vectors (Value). Specifically, for the input multi-modal encoding feature sequence, the query matrix Q, key matrix K, and value matrix V are obtained through linear transformation, then the attention score is calculated, and finally the value matrix is weighted and summed according to the attention score to obtain the output of the self-attention mechanism. In this way, the Transformer can globally associate features of different modalities at each layer: for example, it can focus on the correlation between features of a specific region in the image and force signals at a certain time, or the association between a word in the language instruction and the target object observed in the current vision, thereby fusing these cross-modal associations in the internal hidden state.

[0028] The multi-head attention mechanism captures the feature correlation in different subspaces through multiple attention heads in parallel, so that the fusion result is more comprehensive, that is, the multi-head attention mechanism performs the calculation process of the self-attention mechanism multiple times (head times) in parallel, each time uses a different linear transformation matrix to obtain different Q, K and V, so as to pay attention to the features from multiple different subspaces. The output of each head is spliced and a linear transformation is performed to obtain the final output of the multi-head attention mechanism.

[0029] After being processed by the multi-head attention mechanism, the output is input into the feedforward neural network for nonlinear transformation. At the same time, the input and the output of the feedforward neural network are added in a residual connection manner to alleviate the gradient vanishing problem and accelerate the training of the model. After multiple layers of such processing, the Transformer encoder outputs the fused multi-modal fusion features, that is, the context feature vector sequence, each vector fusing key information from different sources such as vision, force sense, point cloud and language, and retaining the dynamic change clues of the work environment in the time dimension. It should be noted that the specific fusion process can refer to the application of the Transformer encoder in the prior art, which will not be described here. The output of the Transformer provides a unified and semantically rich representation of the environment and task of the robot in the current state.

[0030] In the multi-modal feature fusion, the introduction of the position encoding adds the time step information to the multi-modal features, so that the model can better understand the sequence and correlation of the features in the time sequence. Through the position encoding, the model can more accurately capture the time sequence dependency, thereby improving the quality of the fused features. The self-attention mechanism in the Transformer encoder enables the model to automatically focus on important parts of different modal features and mine the internal relationship between the features. The multi-head attention mechanism pays attention to and fuses the features from multiple different subspaces, further enriching the representation of the features, so that the fused multi-modal features can more comprehensively and accurately reflect the complex information of the original data, and have higher flexibility and effectiveness compared with the traditional feature fusion method.

[0031] S3, inputting the constructed work task knowledge base and the multi-modal fusion features into a large language model to obtain a decomposition result of the target work task, and introducing a human-in-the-loop mechanism to optimize the decomposition result to obtain a subtask sequence; In an embodiment, the step S3 comprises: Obtaining live-line work specification data and live-line work expert data, and extracting work action static association information and work environment dependency rule information therefrom to construct the work task knowledge base; transformer architecture, to decompose the target job task in combination with the job task knowledge base, to obtain an initial subtask sequence; According to the job task knowledge base, the initial subtask sequence is sorted, and the sorted initial subtask sequence is parameterized to generate a first optimized subtask sequence; Based on the physical motion range of the robot, and in combination with the job task knowledge base, the first optimized subtask sequence is optimized to obtain a second optimized subtask sequence; Receive external instructions to introduce the human-in-the-loop mechanism to re-optimize the second optimized subtask sequence to obtain the final subtask sequence.

[0032] Specifically, in complex scenarios such as live-line work, a high-level job task usually needs to be decomposed into multiple manageable subtasks. For example, for a typical job such as "live-line connection and drainage line", the robot cannot complete all steps at once and must follow the process of human-device interaction to execute step by step. The present application utilizes the powerful natural language understanding and reasoning ability of large language models (LLM), combined with the experience of power operation experts and industry expertise, to automatically decompose complex task semantics into several subtasks. The specific steps include: Collect standardized process documents (such as "Live-line Work Safety Regulations") and their operation videos in the power industry as live-line work specification data, and collect experience data of live-line work experts in distribution network live-line work as live-line work expert data. For example, the process provided by experts usually includes tool inspection, target positioning, line clamp installation, and result verification steps. Then use natural language processing technology and rule engine to extract the sequence of each live-line work step, operation requirements, and other job action static correlation information from the collected data, and extract the precautions for work in different weather and terrain conditions, inspection methods for special equipment, and other job environment dependent rule information from the expert data; Finally, based on this information, a job task knowledge base is built, i.e. the extracted information is stored in a structured way in a database, and a relational database (such as MySQL) is used for management. For example, create a "replace drainage line clamp" to store detailed information for each drainage line clamp replacement step, including step number, step name, operation content, required parameters, etc.; create a "step association table" to record the sequence and dependency relationship between steps; create an "environment rule table" to store precautions for work in different weather and terrain conditions, inspection methods for special equipment, etc.

[0033] A knowledge graph is constructed using a graph database, with nodes representing job actions, environmental factors, etc., and edges representing their relationships; these data are structured into a domain knowledge base, which can be in the form of a knowledge graph or a database, recording information such as task descriptions, step sequences, tool requirements, and safety specifications. For example, the knowledge base specifies that "replacing a drainage clamp" requires insulated gloves and a special wrench, and ensures insulation performance in a high-voltage environment. Experts also manually annotate the subtask sequences of typical tasks, define the input (such as tools and environmental conditions), output (such as completion status), and constraints (such as safety requirements and robot movement range) for each subtask, providing high-quality training and reference data for the LLM.

[0034] The target task of the live-line work of the distribution network is clearly expressed in natural language, for example, "complete a certain live-line work operation in a certain distribution line scenario" is converted into an easy-to-understand sentence such as "the robot needs to replace the drainage clamp in the 10kV distribution line", which provides a basis for subsequent processing. This conversion process can use a trained neural network model that has the ability to extract key information from the target task and summarize it. Other existing technologies can also be used for conversion, which will not be described in detail here. Next, the LLM uses the understanding ability of the pre-trained large language model based on the Transformer architecture, combines with the job task knowledge base, analyzes the semantic structure and multi-modal fusion features of the input target work task, identifies key actions (such as "replace"), objects (such as "clamp"), and logical sequences to decompose the task and obtain the initial subtask sequence, i.e. the LLM maps the task to the standard process in the knowledge base through template matching, for example, "replace the drainage clamp" is matched to the template "tool preparation → positioning → installation → inspection". For non-standard or complex tasks, the LLM generates an initial subtask sequence using chain reasoning, for example, deducing that "installing a new clamp" requires "positioning the clamp" to be completed first, and "preparing tools" before positioning. During the decomposition process, the LLM embeds the robot's capabilities and safety constraints to ensure that each subtask is executable. For example, the "position and grasp the target clamp" subtask limits the movement range of the robotic arm to avoid touching the high-voltage line, thereby meeting the safety specifications.

[0035] After generating the subtask sequence, optimization is needed to ensure execution efficiency and logic. The LLM arranges the subtask order according to the job task knowledge base and the logical dependencies and priorities derived from expert rules, such as "detect and prepare job tools" must precede "locate and grasp target clamp" to ensure tool availability. For each ordered subtask, parameterization is performed, i.e., specific execution parameters are defined, such as target coordinates and grasping force for "locate and grasp target clamp" and installation angle and torque for "install new clamp". These parameters are determined based on the job task knowledge base and robot hardware specifications (such as force control accuracy of the robotic arm), and a first-optimized subtask sequence is generated. Subsequently, the feasibility of the subtask sequence is verified through simulation or historical data, such as checking whether "install new clamp" is within the robotic arm's workspace and meets safety distance requirements. If problems are found, the LLM will adjust the subtask definition or order.

[0036] Based on factors such as the physical motion range of the robot (such as maximum travel speed, turning radius, operating range of the robotic arm), safety distance (>0.2m) constraints, mechanical structure limitations, and task logic rules in the job task knowledge base, the first-optimized subtask sequence is further optimized, such as the robotic arm cannot reach a certain position (does not meet safety distance constraints or exceeds physical motion range). The execution method or order of the subtask is adjusted according to the job task knowledge base, robot hardware specifications, and constraint conditions to ensure that the robot can successfully complete each subtask, resulting in a second-optimized subtask sequence.

[0037] The second-optimized subtask sequence is displayed to the operator or expert, and external instructions and feedback information are received from them. The operator or expert can make modification suggestions based on their experience and actual situation, such as adjusting the order of subtasks, adding or deleting certain subtasks, modifying the parameters of subtasks, etc. The second-optimized subtask sequence is further optimized according to the received external instructions to obtain the final subtask sequence, which not only meets the requirements of the robot's capabilities and job specifications, but also fully considers human professional knowledge and actual needs, with higher feasibility and effectiveness.

[0038] The application utilizes the powerful language understanding and generation capability of a large language model to intelligently decompose a task according to a knowledge base and multi-modal information, fully utilizes the advantages of multi-modal data and a language model, and improves the accuracy and rationality of task decomposition; a multi-layer optimization mechanism is used to process an initial subtask sequence: sorting and parameterization processing are performed according to a job task knowledge base, so that the logical order and parameter settings of the subtasks meet the specifications and actual requirements; optimization is performed in combination with the physical motion range of a robot, so that the subtask sequence can be executed within the capability range of the robot; a human-in-the-loop mechanism is introduced for re-optimization, so as to fully utilize the subjective initiative and professional knowledge of a human being, and further improve the quality and feasibility of the subtask sequence; the mechanism can perfect the subtask sequence from different angles, and ensures that the final obtained subtask sequence can efficiently and safely complete a target job task.

[0039] In an embodiment, the target job task is converted into natural language and input into a pre-trained large language model based on a Transformer architecture together with the multi-modal fusion features, to decompose the target job task in combination with a job task knowledge base, and obtain an initial subtask sequence, including: The target job task is parsed to obtain a structured natural language task, which is processed by a feature alignment embedding method together with the multi-modal fusion features, and time sequence position encoding is embedded in the processing result to obtain a cross-modal fusion feature vector containing time sequence information; Based on the cross-modal fusion feature vector and the structured natural language task, knowledge is matched from the job task knowledge base to obtain static knowledge matched with the target job task; The cross-modal fusion feature vector, the structured natural language task and the static knowledge are input into the pre-trained large language model based on the Transformer architecture for dynamic template matching and chain reasoning, to decompose the target job task through a template self-adaptation mechanism in a first path to generate a preliminary subtask framework, and decompose the target job task through a constraint-guided thinking chain in a second path to generate a subtask logic chain; The preliminary subtask framework and the subtask logic chain are subjected to conflict detection and feasibility checking, and after the conflict detection passes and the feasibility checking passes, the initial subtask sequence is generated according to the preliminary subtask framework and the subtask logic chain.

[0040] Specifically, the present application extracts the core information in the target task (oral expression such as "replace the loose drain wire clamp on the 10kV line") by designing a task element extraction template, which includes action type (such as replacement, maintenance, installation, etc.), target object (such as drain wire clamp, insulator, etc., and combined with image recognition results to correct ambiguity, for example, to distinguish between wire clamp and insulator), and constraint conditions (such as 10kV line, live-line work, etc.), and fills the extracted information into a standardized natural language template to generate a structured natural language task, for example: "In the 10kV live-line distribution line, perform the replacement operation on the loose drain wire clamp, and meet the safety distance ≥ 0.2m, environmental wind speed ≤ 5m / s". The structured natural language task and the multi-modal fusion feature are converted into a 768-dimensional text feature vector using the text encoder of LLM (such as Token Embedding layer of LLaMA-3), and the 1024-dimensional multi-modal fusion feature is reduced to 768-dimensional using a multi-layer perception (MLP) to match the dimension of the text feature vector; the association weight between the text and the multi-modal fusion feature is calculated through the modal attention mechanism (such as the weight value of "wire clamp position" in the visual feature and "replace the wire clamp" in the text), and the weighted fusion obtains a 1536-dimensional cross-modal feature vector, and the time position encoding is embedded in the feature (the same as the previous position encoding), and the timestamp encoding is added in the cross-modal feature for dynamic work scene (such as wire shaking), so that the LLM can identify the difference between "the current position of the wire clamp" and "the position 10 seconds ago", and then obtain the cross-modal fusion feature vector containing time sequence information.

[0041] Among them, the modal attention mechanism calculation includes: projecting the text feature and the multi-modal fusion feature into the same dimensional space through linear transformation; generating query vector Q, key vector K and value vector V of the text feature and the multi-modal fusion feature respectively, calculating the similarity between the text query vector and the multi-modal key vector through dot product operation to obtain the original attention score; applying the Softmax function to the original attention score to convert the score into a probability distribution; according to the normalized weight, the multi-modal value vector is weighted and summed to generate the text-guided context feature, and the text feature, the multi-modal feature and its context feature are spliced and averaged or maximized, and the fused feature representation is obtained.

[0042] In addition, a pre-trained feature alignment model can also be used to map the structured natural language task and the multi-modal fusion feature to the same feature space, such as the explicit sequence of disassembling the old clamp and installing the new clamp, adding time position encoding to each feature element, for example, the feature element related to disassembling the old clamp is marked as time sequence 1, and the feature element related to installing the new clamp is marked as time sequence 2, and the cross-modal fusion feature vector containing time sequence information can also be obtained.

[0043] With the cross-modal fusion feature vector and the structured natural language task as the query condition, the similarity between the structured natural language task and the features in the knowledge base is calculated by cosine similarity to find the most matching static knowledge in the knowledge base with the target task; also, through the environmental parameters (such as wind speed 3 m / s) and target attributes (such as clip model) in the cross-modal features, applicable rules are matched from the dynamic rule base as pre-constraints for LLM reasoning.

[0044] Through the target detection model (such as YOLOv8), the target attributes (clip position coordinates, whether blocked, relative distance to the robot arm) are extracted from the visual features in the cross-modal fusion features, and the Fourier transform is performed on the time sequence features in the cross-modal fusion features to identify abnormal patterns (such as torque fluctuation > 5 N·m and duration > 2 s to determine corrosion jamming), and for standard tasks (that is, tasks with fixed operation procedures, the division of tasks can also be based on the relationship between their complexity and the complexity threshold, such as complexity exceeding 0.8 for non-standard tasks, and others for standard tasks, and the task complexity is determined based on the completion time of historical tasks or obtained through linear regression model, decision tree / random forest, neural network, etc.), a standard procedure template library is constructed, which is stored in a three-dimensional structure of task type-core step-variable parameter, such as the basic template for "changing drainage clip": { Task Type: Change Drainage Clip, Core Steps: [Tool Preparation, Target Positioning, Old Part Disassembly, New Part Installation, Quality Inspection], Variable Parameters: { Tool Preparation: {Tool List: [Insulating Wrench, Clamp], Insulation Level: ≥10kV}, Target Positioning: {Accuracy Requirement: ±5cm, Safety Distance: ≥0.2m}, Old Part Disassembly: {Default Torque: 10N·m, Maximum Torque: 15N·m} }, Trigger Conditions: { / / Feature thresholds for triggering step adjustments Robot Arm Extension Limited: {Clip Height: > Maximum Robot Arm Extension Height}, Old Part Rust: {Torque Fluctuation: > 5N·m and Duration: > 2s} } }。

[0045] The parsed features and the templates are matched by a pre-training large language model based on a Transformer architecture, BERT or its improved model, and then the template decomposition task is obtained based on the matching; however, it should be noted that the template is not fixed, but is adapted through a template adaptation mechanism, i.e., the LLM dynamically adjusts the step details based on the cross-modal features according to the task knowledge base and the matching obtained static knowledge, such as the cross-modal fusion feature vector shows that the "clip position is high and the mechanical arm is limited in stretching", so the "adjusting the robot base position" sub-step is added in the "positioning" step, such as showing "old clip rust", so the "increase torque to 15N・m" parameter is added in the "disassembly" step, etc., to decompose the target task based on the template adaptation mechanism in the first path, and generate a preliminary sub-task framework containing dynamically adjusted step details.

[0046] In addition, in the first path, the cosine similarity algorithm can also be used to calculate the matching degree of the parsed multi-modal features and the "trigger condition" in the template, and for the trigger condition with a matching degree ≥0.8, the template adaptation mechanism is started to adjust the steps, and the specific adjustment content can be determined according to the cross-modal features based on the task knowledge base and the matching obtained static knowledge, such as for the mechanical arm stretching limitation: inserting the "adjusting the robot base position" sub-step before the "positioning target" step, and supplementing the parameters (such as "base elevation 0.5m, ensuring that the vertical distance between the mechanical arm and the clip is ≤3.0m"); for "old part rust": modifying the parameters in the "disassembling old part" step (such as "initial torque 12N・m, increasing by 1N・m every 3s, and the maximum is not more than 15N・m"), etc., and integrating the adjusted steps and parameters to generate a sub-task sequence with dynamic details.

[0047] For non-standard tasks (other tasks in addition to standard tasks), the constraint conditions in the task are first quantified, i.e., the safety specifications and the robot capabilities are converted into calculable constraint values, such as the maximum stretching radius of the mechanical arm 1.5m, the operation range of the insulated bucket ±0.8m, and the prohibition of touching live wires (distance ≥0.2m); then the cross-modal fusion features, unstructured natural language tasks, and matching obtained static knowledge are input, and a pre-training large language model based on a Transformer architecture, such as Transformer, is used to generate sub-tasks by using its self-attention mechanism through problem decomposition→step-by-step deduction→self-checking: First round reasoning (target disassembly): To replace the clip blocked by the tree branch, the tree branch needs to be removed first, but it needs to be ensured that the live wire and the clip are not touched. Since the tree branch diameter 8cm < the maximum capacity of the insulated scissors 10cm, it can be directly cut.

[0048] Second round reasoning (step refinement): Step 1: Move the insulated bucket to 1 m next to the branch (due to the insulated bucket's movement range ±0.8 m, the current distance from the target is 2 m, so it needs to be moved twice, each time 0.8 m and 0.2 m, to avoid exceeding the range; Step 2: Adjust the angle of the mechanical arm so that the insulated scissors are aligned with the branch (the cutting point is selected 30 cm away from the wire clamp to avoid the wire clamp and ensure that there is no residual obstruction after cutting); Step 3: Cut the branch (force 50 N, as the diameter is 8 cm, ensure that it is cut off, and monitor the reaction force to avoid the cutting edge jamming); Step 4: Confirm that the branch has fallen (visually check that there is no residual obstruction and that the wire has not been touched) → follow the regular replacement steps.

[0049] Third round of reasoning (parameter verification): Verify the mechanical arm extension distance in Step 2: The current distance between the insulated bucket and the branch is 1 m, and the mechanical arm needs to be extended by 0.8 m to reach it (1 m + 0.8 m = 1.8 m < maximum radius 1.5 m? No, calculation error! Recalculate: move the insulated bucket to a distance of 0.7 m from the branch, extend the mechanical arm by 0.8 m, the total distance is 1.5 m = maximum radius, which meets the constraints.

[0050] Then check the dependency between steps through the LLM's self-recursive mechanism (such as "cutting the branch" must be after "moving the insulated bucket", etc.), and after the logical coherence detection passes, finally generate a sub-task logic chain containing operation details, parameter constraints, and logical order.

[0051] Check the preliminary sub-task framework and sub-task logic chain to determine whether there are any time conflicts (such as multiple sub-tasks not requiring simultaneous execution), resource conflicts (such as the robot not using multiple non-shareable devices simultaneously), and operation conflicts (such as the operation sequence meeting the specifications). If not, assess whether each sub-task can be executed in the actual work environment by considering the robot's physical movement range, such as whether all sub-tasks are within the robot's reachable range. If the conflict detection passes and the feasibility check passes, integrate the preliminary sub-task framework and the sub-task logic chain to generate an initial sub-task sequence. If there are conflicts or unfeasible situations, return to adjust the parameters of the template adaptive mechanism or the constraint-guided thinking chain to re-decompose the task.

[0052] The invention adopts a dual-path (template adaptive mechanism and constraint-guided thinking chain) approach to decompose the target task, ensuring the normativity and consistency of decomposition (through template adaptation) and allowing flexible reasoning based on specific task conditions (through constraint-guided thinking chain), thereby improving the accuracy and adaptability of task decomposition.

[0053] Model training is the core of achieving high-quality task decomposition. The training dataset is composed of expert-labeled subtask sequences and corresponding operation data (e.g., visual images, force feedback, and motion trajectories). For example, videos, sensor data, and operator instructions from multiple “replace drainage clamp” operations are collected. Supervised fine-tuning (SFT) is used to enable the LLM to learn the expert’s decomposition logic, input a task description, and output a subtask sequence and its parameters. To further optimize the decomposition strategy, reinforcement learning (RL) can be used with task completion efficiency and safety as the reward function. For example, if the robot fails to perform “install new clamp” (e.g., the clamp is loose), the model will adjust the decomposition strategy by adding a “force calibration” subtask. Considering the common tasks of live-line work (e.g., tool inspection and target positioning), a general model can be pre-trained through transfer learning, and then fine-tuned for specific tasks (e.g., clamp installation) to improve the model’s generalization ability and training efficiency.

[0054] For example, the process of “replace drainage clamp” usually includes steps such as tool inspection, positioning operation, installation, and verification, which are provided by power operation and maintenance experts. The specific implementation process is as follows: input the task description “replace drainage clamp in 10kV distribution network line” and environmental information (e.g., line location and weather conditions), the LLM extracts key information (target: clamp, action: replace, environment: high-voltage line), and generates a subtask sequence: the first subtask “inspect and prepare work tools” calls the VLA model to verify the integrity of the insulating gloves and special wrench; the second subtask “locate and grasp the target clamp” processes visual data through the VLA model to generate a grasping action; the third subtask “install new clamp” controls the robot arm to perform installation with a specified torque; and the fourth subtask “final check” uses the vision system to confirm the installation quality. After each subtask is executed, the VLA model provides feedback (e.g., success rate of grasping and stability of installation), and the LLM optimizes subsequent actions based on the feedback. If the operator finds that the clamp model is incorrect, he can input “replace model Y clamp”, and the LLM will re-decompose the task and insert a “replace clamp model” subtask.

[0055] In an embodiment, the receiving external instructions to introduce the human-in-the-loop mechanism to re-optimize the secondary optimization subtask sequence to obtain a final subtask sequence comprises: receiving external instructions and standardizing the external instructions to obtain standardized instruction triplets containing type, content, and timestamp; based on the double-threshold checking rule constraint, inputting the standardized instruction triplets into a work pre-play model constructed based on digital twin technology for simulation to generate an effective instruction queue, and judging the type of the effective instruction queue; when the effective instruction queue is a priority intervention instruction, analyzing logical relationships between subtasks contained in the secondary optimization subtask sequence by a large language model, and adjusting an execution position of a target subtask corresponding to the effective instruction queue to obtain a first subtask sequence; when the effective instruction queue is a task replacement instruction, generating a breakpoint mark based on a target subtask corresponding to the effective instruction queue, and calling the visual language action model to generate a new task according to the breakpoint mark to adjust the secondary optimization subtask sequence to obtain a second subtask sequence; when the effective instruction queue is a parameter correction instruction, performing parameter boundary checking on the effective instruction queue, and converting the effective instruction queue into a control signal after passing the checking to correct the secondary optimization subtask sequence to obtain a third subtask sequence.

[0056] Specifically, the application receives external instructions and standardizes them to convert them into a standardized instruction triple including instruction type: priority intervention / task replacement / parameter correction, core content: specific operation description, and timestamp: time of reception.

[0057] Based on the double-threshold checking rule constraint that directly intercepts the instruction and feeds back the hard constraint of "the instruction violates the safety distance, please re-input" if the rehearsal result violates the safety specification (such as touching a live body), and prompts the operator "the instruction may prolong the operation time, do you continue?" if the rehearsal result may reduce efficiency (such as frequent adjustment leading to task timeout), and waits for confirmation to execute the soft constraint of time, the standardized instruction triple is input into the operation rehearsal model (including real-time three-dimensional modeling of robots, wires, and environment) containing the target operation scene constructed by digital twin technology to simulate the state after the instruction execution, and an effective instruction queue passing the safety check and sorted by priority is output, and the type of the effective instruction queue is judged.

[0058] When the effective instruction queue is a priority intervention instruction (such as "check tool insulation first, then relocate the clamp"), the instruction and the secondary optimization subtask sequence (original sequence: positioning → tool checking → installation) are input, the logical dependency between subtasks (such as "installation" must be completed after "positioning", and "tool checking" and "positioning" are independent and can be interchanged) is analyzed by a large language model LLM, the target subtask (tool checking) in the priority intervention instruction is moved to a priority position, if there are pre-requisite steps in the original sequence (such as "tool checking" requires "taking out the tool" first), the pre-requisite steps are automatically completed, a new sequence is generated and output: take out the tool → tool checking → positioning → installation, that is, the first subtask sequence after adjustment and retaining logical dependency.

[0059] When the valid instruction queue contains a task replacement instruction (such as "pause installation and clear branches next to the wires first"), the state data corresponding to the instruction and the secondary optimized sub-task sequence (such as the installation step being 30% complete and the current pose of the robotic arm) are used as input. The progress parameters of the current sub-task (such as the installation torque value and the robotic arm joint angle) are recorded, and a breakpoint marker is generated (such as {task ID: install wire clamp, progress: 30%, recovery condition: branch clearing completed}). Based on the breakpoint marker, the VLA model is called to generate the sub-steps of the new task (clearing branches) (locating branches → adjusting the position of the insulating bucket → controlling the insulating shears to cut), and the new task path is ensured to be smoothly connected with the current robotic arm pose (minimizing the movement distance), resulting in a new sub-task sequence with breakpoint markers.

[0060] When the effective instruction queue contains a parameter correction instruction (e.g., "clamping force reduced from 8N ​​to 6N"), this instruction and the parameter execution range of the secondary optimization sub-task sequence (e.g., "drainage clamp clamping force safe range 5-10N") are used as input. First, the effective instruction queue is checked for parameter boundaries, confirming that the correction value (6N) is within the safe range (5-10N). If it exceeds the range (e.g., input 12N), it automatically clamps to the maximum value (10N) and prompts "Parameter exceeds safe range, adjusted to 10N." The correction parameter in the parameter correction instruction is converted into a robotic arm control signal (e.g., current instruction), and the actual clamping force is monitored in real time by a force sensor (feedback every 10ms) to ensure the deviation from the correction value is ≤0.5N. The correction instruction and robotic arm control signal are integrated into the secondary optimization sub-task sequence to obtain the third sub-task sequence with updated parameters. The sub-task sequence optimized by external instructions is the final sub-task sequence.

[0061] This invention unifies external instructions into triples containing type, content, and timestamp, solving the problem of format differences between instructions from different sources and providing standardized input for subsequent processing; it constructs a job pre-simulation model to simulate the impact of instructions on the task sequence, identify potential conflicts in advance, and avoid errors in actual execution; it combines the advantages of digital twin (physical simulation), LLM (logic analysis), and VLA (action generation) to form a "perception-decision-execution" closed loop, significantly improving the flexibility and accuracy of task optimization.

[0062] S4. Based on the sub-task types in the sub-task sequence, the sub-task sequence and the multi-source modal data are input into a visual language action model or a reinforcement learning model for processing to generate motion instructions corresponding to the sub-task sequence, so that the robot can start the execution process of the target task; wherein, the visual language action model includes a visual network layer, a language network layer and a flow matching action network layer; the multi-source modal data also includes real-time sensor data; In one embodiment, step S4 includes: judging a subtask type in the subtask sequence, if a subtask in the subtask sequence is determined to be of a first type, inputting the multi-source modality data into the reinforcement learning model for processing to generate a sub-motion instruction corresponding to the first type of subtask; if a subtask in the subtask sequence is determined to be of a second type, inputting the multi-modal fusion feature and the second type of subtask into the visual language action model, so that the visual network layer identifies the multi-modal fusion feature to obtain a target visual feature corresponding to the second type of subtask, so that the language network layer combines the job task knowledge base to analyze the second type of subtask to obtain an ordered subtask sequence, and so that the flow matching action network layer processes the target visual feature, the ordered subtask sequence and the real-time sensor data to generate a sub-motion instruction corresponding to the second type of subtask; integrating each of the sub-motion instructions to obtain a motion instruction corresponding to the subtask sequence, so that the robot starts an execution process of the target job task.

[0063] Specifically, after task decomposition, the visual language action model (VLA) or the reinforcement learning model is used according to the type of the subtask to realize action execution, and the specific steps include: judging the type of a subtask contained in the finally obtained subtask sequence; if a subtask in the subtask sequence is a fixed subtask such as grabbing a wire or screwing, it is of a first type, and a pre-trained reinforcement learning model is called to process multi-source modality data to generate a sub-motion instruction (joint speed / torque) corresponding to the first type of subtask. These reinforcement learning (RL) models are different in type, but they are all constructed for a single subtask (such as screwing) to realize efficient and robust robot autonomous operation in a specific task scenario. These skill models are equivalent to expert-level low-level control strategies. When the robot needs to perform a certain specific subtask, the corresponding skill model can be called to complete it. For example, for a subtask such as "grabbing a wire" that repeatedly appears and has a relatively fixed operation mode, a reinforcement learning model is trained to adjust the posture of a manipulator to stably hold a vibrating wire.

[0064] If the subtask corresponding to a step in the subtask sequence is identified as a first type subtask, which has an available skill model, the multi-source modality data is input into the skill model to obtain an action decision. The multi-source modality data includes visual observations (target object image features) related to the subtask, the current pose of the robot end, force sensation readings, and other necessary state variables. The skill model takes these states as input, and calculates the output low-level motion control instructions (including at least velocity adjustment or torque instructions for driving specific robot joint movements to achieve its own position adjustment) through its internal policy network. Since these models have been trained a large number of times and optimized for the corresponding scene, they can execute the corresponding operation with higher efficiency and success rate than general decision algorithms. For example, the RL skill model may have learned to optimize the sequence and force of screwing a bolt, so that it can be completed faster and without damaging the threads than a fixed strategy set by a human.

[0065] During the training phase of the skill model, a reinforcement learning algorithm is used to allow the robot agent to repeatedly interact with the environment in a simulated environment, starting from zero experience to explore the optimal strategy. A reasonable reward function is set during the training process to guide learning, for example, giving a positive reward for successfully completing a subtask, and a negative reward for failure or timeout, and the time and energy consumption can also be included in the reward function to balance. Through such trial and error learning, the RL agent gradually adjusts its policy network parameters to maximize the cumulative expected reward, that is, to gradually approach the optimal strategy. However, random exploration alone may take a long time to converge, so to improve training efficiency and avoid unnecessary collision risks, expert demonstration data is introduced for imitation learning at the beginning of training. Specifically, the robot imitates a number of demonstration trajectories of a human or a traditional controller completing a subtask, and converts these trajectories into training samples for the policy network, so that the agent starts learning from an initial strategy with higher performance. This can significantly speed up the convergence of reinforcement learning, while ensuring the reasonableness of the strategy. Then gradually transition to pure reinforcement learning to allow the model to fine-tune and optimize itself on the basis of what has been achieved. The final skill model reaches a level close to or even exceeding that of a human expert in its specialized task, with high accuracy, fast response, and strong robustness.

[0066] If a subtask in the subtask sequence is a non-fixed subtask, it is a second type subtask, and the multi-modal fusion features and the second type subtask are input into the visual language action model VLA for processing to obtain a sub-movement instruction (joint velocity / torque instruction corresponding to a collision-free smooth trajectory) corresponding to the second type subtask.

[0067] The visual language action model processes the input multi-modal fusion features and the second type of subtask through a visual network layer (CLIP-ViT) to identify the target visual features, i.e., the target position and safe distance, for output. Specifically, the visual network layer is responsible for processing environmental images and sensor data to extract spatial information and geometric features of obstacles and target objects (such as wires, clips, and wire strippers). The CLIP-ViT-L-336px is used to process high-resolution RGB images (from Intel RealSense or similar cameras), depth images (from lidar or depth sensors), and point cloud data (generated by 3D sensors) after feature extraction and fusion to form multi-modal fusion features. For live-line work scenarios, the visual network layer pays special attention to the minimum safe distance of the wire (usually >0.2m) and the precise position of the target object (x, y, z coordinates). For example, in a wire stripping task, the visual network can identify the diameter and position of the cable and the clamping point of the wire stripper to ensure that the action generation meets the insulation and safety constraints.

[0068] The second type of subtask is parsed into an action sequence through a language network layer (LLaMA-3.1), and constraint conditions are generated by combining the task knowledge base to obtain an ordered subtask sequence. The language model combines with an expert knowledge base (such as artificial live-line work processes and safety specifications) to ensure that the subtask sequence meets the standards of the power industry (such as avoiding touching live components and ensuring strict action sequences). In addition, the model supports a "man-in-the-loop" mechanism that allows operators to dynamically adjust tasks through natural language instructions (such as "adjust the wire stripper angle").

[0069] The target visual features, ordered subtask sequence, and real-time sensor data are input into the flow matching action network layer to generate split motion instructions corresponding to the second type of subtask through flow matching algorithms for action prediction (a flow matching network based on the Transformer architecture generates action trajectories) and safety verification (checks whether the torque exceeds the knowledge base threshold, etc.). Flow matching gradually generates smooth trajectories that meet environmental constraints from random noisy trajectories through iterative denoising. The generation process is conditioned on visual features (obstacle positions, target positions) and sensor data (such as wind speed, wire sway) to ensure collision-free trajectories and meet safe distance requirements (e.g., avoid wires by more than 0.2m). The action expert adjusts motion parameters (such as closing speed, force threshold) based on feedback to ensure gentle grasping without damaging the target. For example, when the tactile sensor detects that the grasping force reaches a stable threshold (such as 10N), the action expert immediately stops applying greater pressure.

[0070] The outputs of the two models are time-synchronized by a TimeSynchronizer node of ROS (Robot Operating System) to align the instruction time stamps, and then it is checked whether the robot arm simultaneously receives the conflicting instructions of "moving to B point" and "moving to A position" to perform conflict detection, and the instructions passing the detection are fused in sequence to generate a global motion instruction sequence, i.e., a motion instruction corresponding to the subtask sequence, so that the robot starts the execution process of the target operation task. It should be noted that the specific process of data processing of some algorithms (flow matching) and models (LLaMA-3.1, etc.) in the above scheme can refer to the application of the algorithm and model in the prior art, and will not be described here.

[0071] In the embodiment, a distributed deep learning framework PyTorch is adopted, and GPU cluster is used to accelerate training. By designing a loss function (including cross-entropy loss, mean square error loss, and inter-modal contrast loss, etc.), the performance of the model is effectively evaluated during the training process. In addition, through hyperparameter tuning (including learning rate, dropout, batch size, etc.), the training time is shortened and the model performance is improved. Finally, the multi-dimensional indicators such as accuracy and response speed are used to comprehensively evaluate the multi-modal large model. Combined with the simulation site test data and the simulation platform verification result, the model structure and parameters are continuously iterated and optimized, and the adaptability and decision accuracy of the model in the complex dynamic live working environment are continuously improved, so as to ensure the efficiency and safety of the robot in actual operation.

[0072] The application breaks through the limitation of traditional single model processing all tasks, automatically selects a reinforcement learning model or a visual language action model VLA according to the characteristics of the subtask, significantly improves the adaptability and efficiency of task processing, improves the smoothness and safety of trajectory generation, ensures the stability and repeatability of the robot in the dynamic environment, and meets the insulation and distance requirements of live working; the reinforcement learning model for a single subtask can optimize the strategy through interaction with the environment, introduce expert demonstration data to accelerate the learning process, and significantly improve the adaptability and decision accuracy of the robot in the complex dynamic environment.

[0073] The embodiment of the application is based on how to improve the working efficiency of the existing autonomous power distribution live working robot, and designs a robot control method based on a multi-modal large model. The method acquires multi-source modal data of a scene where a target working task is located in real time, and extracts features from different types of data by using corresponding machine learning models. This multi-source data fusion processing method can more comprehensively and accurately capture scene information, and provides a richer feature basis for subsequent task decomposition and instruction generation compared with single modal data processing, thereby improving the understanding ability of complex working scenes. After position encoding and Transformer fusion processing of the multi-modal features to obtain multi-modal fusion features, a working task knowledge base is constructed, and the fusion features are input into a large language model to decompose the task, and a human-in-the-loop mechanism is introduced to optimize the decomposition result. This combination fully utilizes the language understanding and reasoning ability of the large language model and the professional knowledge and experience of humans, so that the task decomposition is more in line with actual needs and logic, and the accuracy and rationality of the task decomposition are improved. Based on the types of subtasks in the subtask sequence, a visual language action model or a reinforcement learning model is flexibly selected for processing to generate motion instructions. This dynamic model selection mechanism can take advantage of different models according to different subtask characteristics, improve the efficiency and accuracy of instruction generation, better control the robot to execute complex working tasks, and further improve the working efficiency of the existing autonomous power distribution live working robot.

[0074] It should be noted that although each step in the above flowchart is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps has no strict order limitation, and these steps can be executed in other orders.

[0075] In another embodiment, as shown in FIG. 1, Figure 2 The second aspect of the application provides a robot control system based on a multi-modal large model, comprising: The feature extraction module 10 is configured to acquire multi-source modal data of a scene where a target working task is located in real time, and extract features from the multi-source modal data by using corresponding machine learning models based on the types of the multi-source modal data, to obtain multi-modal features. The feature fusion module 20 is configured to perform position encoding and Transformer fusion processing on the multi-modal features to obtain multi-modal fusion features. The task decomposition module 30 is configured to input the constructed working task knowledge base and the multi-modal fusion features into a large language model to obtain a decomposition result of the target working task, and introduce a human-in-the-loop mechanism to optimize the decomposition result to obtain a subtask sequence. The instruction execution module 40 is configured to input the subtask sequence and the multi-source modal data into a visual language action model or a reinforcement learning model based on a subtask type in the subtask sequence for processing, to generate a motion instruction corresponding to the subtask sequence, and to enable the robot to start an execution process of the target task.

[0076] It should be noted that the above-mentioned various modules in the robot control system based on the multi-modal large model can be realized by software, hardware, and combinations thereof, in whole or in part. The above-mentioned various modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned various modules. For specific limitations of the robot control system based on the multi-modal large model, refer to the limitations of the robot control method based on the multi-modal large model, both of which have the same functions and effects, and will not be described here.

[0077] The third aspect of the application provides an electronic device, which comprises: a processor, a memory and a bus; the bus is configured to connect the processor and the memory; the memory is configured to store operation instructions; the processor is configured to execute the operations corresponding to the robot control method based on the multi-modal large model according to the operation instructions.

[0078] In an optional embodiment, an electronic device is provided, as shown in Figure 3 The electronic device 5000 shown in Figure 3 The electronic device 5000 shown in

[0079] The processor 5001 can be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure. The processor 5001 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0080] The bus 5002 can include a path that transmits information between the above-described components. The bus 5002 can be a PCI bus or an EISA bus, etc. The bus 5002 can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 3 Only one thick line is used in the middle, but it does not mean that there is only one bus or one type of bus.

[0081] The memory 5003 can be a ROM, or other type of static storage device that can store static information and instructions, a RAM, or other type of dynamic storage device that can store information and instructions, an EEPROM, a CD-ROM or other optical disk storage, a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0082] The memory 5003 is used to store application program codes for implementing the scheme of the present application, and is controlled by the processor 5001 to perform. The processor 5001 is used to execute the application program codes stored in the memory 5003 to realize the content shown in any of the foregoing method embodiments.

[0083] The electronic device includes, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a car terminal (e.g., a car navigation terminal), etc., and a fixed terminal such as a digital TV, a desktop computer, etc.

[0084] The fourth aspect of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the program is executed by a processor to implement the robot control method based on a multi-modal large model shown in the first aspect of the present application.

[0085] Still another embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and when the computer program is run on a computer, the computer can execute the corresponding content in the foregoing method embodiments.

[0086] In addition, an embodiment of the present application also proposes a computer readable storage medium, and the computer readable storage medium stores a computer program, and the program is executed by a processor to implement the steps of the above method.

[0087] To sum up, the application relates to the technical field of robot control, and discloses a robot control method, system, device and medium based on a multimodal large model, which obtains multimodal features through multimodal data of a scene where a work task is located and processing by a machine learning model, performs position coding and Transformer fusion processing, obtains multimodal fusion features, inputs the multimodal fusion features and a constructed work task knowledge base into a large language model to decompose a target work task, introduces a human-in-the-loop mechanism to optimize the decomposition result, and obtains a subtask sequence; according to a subtask type in the subtask sequence, the subtask sequence is processed through a visual language action model or a reinforcement learning model to generate a motion instruction to enable a robot to start an execution process of the target work task; and the work task is processed through the multimodal large model LLM, VLA and the like, thereby improving the work efficiency of the autonomous distribution network live work robot.

[0088] Each embodiment in the specification is described in a progressive manner, and the same or similar parts of each embodiment can be referred to each other, and each embodiment mainly describes the difference from other embodiments. Especially, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts can be referred to the part of the method embodiment. It should be noted that, each technical feature of the above-mentioned embodiments can be combined arbitrarily, in order to make the description simple, each technical feature of the above-mentioned embodiments is not described all possible combinations, however, as long as the combination of these technical features does not exist contradictory, it should be considered that it is within the scope of the specification.

[0089] The above-mentioned embodiments only express several preferred embodiments of the application, the description is more specific and detailed, but it should not be understood as the limitation of the patent scope of the application. It should be pointed out that, for ordinary skilled in the art, without departing from the technical principles of the application, a number of improvements and replacements can be made, and these improvements and replacements should be regarded as the protection scope of the application. Therefore, the protection scope of the patent of the application should be subject to the protection scope of the claims.

Claims

1. A robot control method based on a multimodal large model, characterized in that, include: Multi-source modal data of the scene where the target task is located is collected in real time, and based on the type of the multi-source modal data, the corresponding machine learning model is used to extract features from the multi-source modal data to obtain multi-modal features; The multimodal features are subjected to position encoding and Transformer fusion processing to obtain multimodal fused features; The constructed task knowledge base and the multimodal fusion features are input into the large language model to obtain the decomposition result of the target task. A human-in-the-loop mechanism is introduced to optimize the decomposition result to obtain a sub-task sequence. Based on the subtask types in the subtask sequence, the subtask sequence and the multi-source modal data are input into a visual language action model or reinforcement learning model for processing to generate motion instructions corresponding to the subtask sequence, so that the robot can start the execution process of the target task.

2. The robot control method based on a multimodal large model according to claim 1, characterized in that, The multi-source modal data includes force and point cloud data, image data, and audio data; among which... The step of using a corresponding machine learning model to extract features from the multi-source modal data to obtain multi-modal features includes: The force perception and point cloud data are input into a long short-term memory network model for processing to capture force interaction patterns and obtain temporal dynamic features. The image data is input into an improved ResNet-50 model for processing to obtain local features containing the target object; The audio data is input into the Whisper-Base model and the BERT-Base-Chinese model for processing to obtain speech features used to parse and execute instructions; The multimodal features are obtained by combining the temporal dynamic features, the local features, and the speech features.

3. The robot control method based on a multimodal large model according to claim 1, characterized in that, The step of performing position encoding and Transformer fusion processing on the multimodal features to obtain multimodal fused features includes: Multimodal encoded features are obtained by embedding time steps into the multimodal features through position encoding; The multimodal encoded features are input into the Transformer encoder to perform feature fusion through self-attention and multi-head attention mechanisms to obtain the multimodal fused features.

4. The robot control method based on a multimodal large model according to claim 1, characterized in that, The constructed task knowledge base and the multimodal fusion features are input into a large language model to obtain a decomposition result of the target task. A human-in-the-loop mechanism is then introduced to optimize the decomposition result, resulting in a sequence of subtasks, including: Acquire live-line work specification data and live-line work expert data, and extract static correlation information of work actions and work environment dependency rule information from them to construct the work task knowledge base; The target task is transformed into natural language and input into a pre-trained large language model based on the Transformer architecture along with the multimodal fusion features. This is then combined with the task knowledge base to decompose the target task and obtain an initial sub-task sequence. The initial subtask sequence is sorted according to the task knowledge base, and the sorted initial subtask sequence is parameterized to generate an optimized subtask sequence. Based on the robot's physical range of motion and combined with the task knowledge base, the first-order optimization sub-task sequence is optimized to obtain the second-order optimization sub-task sequence. Receive external instructions to introduce the human-in-the-loop mechanism to further optimize the secondary optimization subtask sequence, thereby obtaining the final subtask sequence.

5. The robot control method based on a multimodal large model according to claim 4, characterized in that, The step involves converting the target task into natural language and inputting it, along with the multimodal fusion features, into a pre-trained large language model based on the Transformer architecture. This decomposes the target task using the task knowledge base to obtain an initial sub-task sequence, including: The target task is parsed to obtain a structured natural language task, which is then processed with the multimodal fusion features using a feature alignment embedding method. Temporal position encoding is embedded in the processing result to obtain a cross-modal fusion feature vector containing temporal information. Based on the cross-modal fusion feature vector and the structured natural language task, knowledge is matched from the task knowledge base to obtain static knowledge that matches the target task. The cross-modal fusion feature vector, the structured natural language task, and the static knowledge are input into the pre-trained large language model based on the Transformer architecture for dynamic template matching and chain reasoning. In the first path, the target task is decomposed through a template adaptation mechanism to generate a preliminary sub-task framework. In the second path, the target task is decomposed through a constraint-guided thinking chain to generate a sub-task logic chain. The preliminary subtask framework and the subtask logic chain are subjected to conflict detection and feasibility verification. After the conflict detection and feasibility verification are passed, the initial subtask sequence is generated based on the preliminary subtask framework and the subtask logic chain.

6. The robot control method based on a multimodal large model according to claim 4, characterized in that, The step of receiving external instructions to introduce the human-in-the-loop mechanism to further optimize the secondary optimization subtask sequence, thereby obtaining the final subtask sequence, includes: Receive external instructions and standardize the external instructions to obtain a standardized instruction triplet containing type, content and timestamp; Based on the dual threshold verification rule constraint, the standardized instruction triplet is input into the job pre-simulation model built based on digital twin technology for simulation, so as to generate a valid instruction queue and determine the type of the valid instruction queue. When the effective instruction queue contains priority intervention instructions, the logical relationship between the subtasks contained in the secondary optimization subtask sequence is analyzed by the large language model, and the execution position of the target subtask corresponding to the effective instruction queue is adjusted to obtain the first subtask sequence. When the effective instruction queue is a task replacement instruction, a breakpoint marker is generated based on the target subtask corresponding to the effective instruction queue, and a new task is generated by calling the visual language action model according to the breakpoint marker, so as to adjust the secondary optimization subtask sequence and obtain the second subtask sequence. When the valid instruction queue contains parameter correction instructions, parameter boundary checks are performed on the valid instruction queue. After the checks pass, the valid instruction queue is converted into control signals to correct the secondary optimization subtask sequence, thereby obtaining the third subtask sequence.

7. The robot control method based on a multimodal large model according to claim 1, characterized in that, The visual-language action model includes a visual network layer, a language network layer, and a flow-matching action network layer; the multi-source modal data also includes real-time sensor data; wherein... The step of inputting the sub-task sequence and the multi-source modal data into a visual language action model or reinforcement learning model for processing, based on the sub-task type in the sub-task sequence, to generate motion commands corresponding to the sub-task sequence, so that the robot can start the execution process of the target task, includes: The type of subtask in the subtask sequence is determined. If a subtask in the subtask sequence is determined to be of the first type, the multi-source modal data is input into the reinforcement learning model for processing to generate sub-motion instructions corresponding to the first type of subtask. If a subtask in the subtask sequence is determined to be of the second type, the multimodal fusion feature and the second type of subtask are input into the visual language action model, so that the visual network layer recognizes the multimodal fusion feature to obtain the target visual feature corresponding to the second type of subtask, so that the language network layer combines the task knowledge base to parse the second type of subtask to obtain an ordered subtask sequence, so that the flow matching action network layer processes the target visual feature, the ordered subtask sequence and the real-time sensor data to generate the sub-motion command corresponding to the second type of subtask; The individual motion commands are integrated to obtain the motion commands corresponding to the sub-task sequence, so that the robot can start the execution process of the target task.

8. A robot control system based on a multimodal large model, characterized in that, include: The feature extraction module is used to collect multi-source modal data of the scene where the target task is located in real time, and extract features from the multi-source modal data using the corresponding machine learning model based on the type of the multi-source modal data to obtain multi-modal features; The feature fusion module is used to perform position encoding and Transformer fusion processing on the multimodal features to obtain multimodal fused features; The task decomposition module is used to input the constructed task knowledge base and the multimodal fusion features into the large language model to obtain the decomposition result of the target task, and to introduce a human-in-the-loop mechanism to optimize the decomposition result to obtain a sub-task sequence. The instruction execution module is used to input the subtask sequence and the multi-source modal data into a visual language action model or reinforcement learning model for processing based on the subtask type in the subtask sequence, and generate motion instructions corresponding to the subtask sequence so that the robot can start the execution process of the target task.

9. An electronic device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the robot control method based on a multimodal large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the robot control method based on a multimodal large model as described in any one of claims 1 to 7.

Images