Live-line work atomic action sequence generation method, system and equipment based on multi-modal instruction and medium

Abstract

The invention discloses a hot-line work atomic action sequence generation method, system and device based on a multi-mode instruction and a medium, and the method comprises the steps: carrying out the hot-line work atomic action sequence generation according to a video and voice intention instruction of an operator and a corresponding robot action sequence based on machine learning; constructing a machine learning task model oriented to a dynamic hot-line work environment, determining a high-quality distribution network hot-line work sample library, carrying out fine tuning training on a power industry model, and carrying out operation classification description based on prompt learning to obtain an operation prompt template; the attention multi-modal model extracts multi-modal features and maps the multi-modal features to an intention instruction mode through thinking chain fine tuning training, and the power industry model generates an atomic action sequence through combination of an intention mode instruction and an operation prompt template, so that the problems that a robot is difficult to understand semantics, low in efficiency, unsmooth in field operation process and poor in robot experience are solved. The operation efficiency is low, and the operation time is long.

Description

Technical Field

[0001] The present invention relates to the field of robots for live-line working, and in particular to a method, system, device and medium for generating atomic action sequences for live-line working based on multi-modal instructions. Background Art

[0002] Due to the compact arrangement of overhead lines in the distribution network, complex and diverse line types, and the high labor intensity of outdoor high-altitude operations, manual live-line operations still have deficiencies in safety and efficiency. With the development of technologies such as machinery, electronics, computers, sensors, and artificial intelligence, live-line operation robots have been widely used to assist or replace manual work in distribution network live-line operations. Compared with manual operations, power robots have the advantages of high operating accuracy, high operating efficiency, and good safety. However, there are still many limitations in practical applications:

[0003] (1) The robot has a low ability to understand the working scene. The working scenes of distribution network robots are usually more complex, including high altitude, narrow, and strong electromagnetic interference environments. The robot has a low ability to understand the complex and changeable environment of distribution lines and non-standard scenes. The existing large amount of working data is not fully utilized, and the robot cannot learn and adapt to the working site environment autonomously.

[0004] (2) Poor understanding of the intention of human-machine collaboration instructions. Due to the deviation of the robot's accuracy in modeling, point selection, and movement, repeated operations or manual adjustments are required. However, the robot has difficulty and low efficiency in understanding the semantics of human-machine collaboration, which makes the on-site operation process not smooth, resulting in low operation efficiency and long operation time.

[0005] (3) It is difficult to optimize real-time actions at the work site. The environment at the robot’s work site is often complex and changeable, including lighting conditions, obstacle distribution, temperature and humidity, and other factors. These factors may affect the robot’s perception and decision-making, resulting in the robot’s inability to adjust and optimize action execution based on instant feedback information to adapt to changes in the on-site work environment. Summary of the invention

[0006] Purpose of the invention: The present invention aims to provide a method for generating atomic action sequences for live working based on multimodal instructions of machine learning and prompt learning by introducing a multimodal thinking chain fine-tuning paradigm; another purpose of the present invention is to provide a system, device and medium for generating atomic action sequences for live working based on multimodal instructions.

[0007] Technical solution: The method for generating live working based on multi-modal instructions of the present invention comprises the following steps:

[0008] Based on machine learning, according to the intention instructions in historical live working tasks and their corresponding robot action sequences, a machine learning task model for dynamic live working environments is constructed to determine the distribution network live working sample library;

[0009] According to the historical distribution network live robot operation process, based on prompt learning, the operation classification description is carried out to obtain the operation prompt template;

[0010] Construct a multimodal model based on attention and conduct fine-tuning training on the thinking chain;

[0011] Build a power industry model and use the distribution network live operation sample library to fine-tune the power industry model;

[0012] Input vision, point cloud, and instructions into the attention multimodal model to obtain the intention modality instruction;

[0013] Input the intention modal instructions into the power industry model, combine them with the operation prompt template, and generate atomic action sequences.

[0014] Furthermore, the steps for determining the distribution network live working sample library are as follows:

[0015] According to historical visual information, historical environmental information, historical work tasks and historical performance indicators, the voice commands and input commands in the historical work tasks are analyzed, and machine learning is used to identify the operator's work actions and the corresponding robot's atomic actions in the historical visual information. Combined with historical environmental information and historical performance indicators, a machine learning task model for dynamic live working environments is constructed based on the working environment, intention commands and the robot's atomic actions to obtain a distribution network live working sample library.

[0016] Furthermore, the attention-based multimodal model includes a visual module, a point cloud module, and a language module; the visual module extracts image features y from the image input vis , the point cloud module extracts point cloud features y from the point cloud point , the language module extracts language features y from the language input ins According to the characteristics of the live task and the distribution of the data, a thinking chain is constructed for the feature extractor, language projection layer and prefix language adapter, and fine-tuning training is performed to adjust the parameters of the attention-based multimodal model.

[0017] Furthermore, the visual, point cloud, and command are input into the attention multimodal model to obtain the intention modal command, as follows: The point cloud feature y is fused through cross attention point , image feature y vis and language feature y ins :

[0018] Use three linear layers to generate Query-Key-Value pairs:

[0019] Q=y point ·W Q

[0020] K=y vis ·W K

[0021] V=y ins ·W V

[0022] Among them, Q, K, and V represent Query, Key, and Value respectively, and W Q , W K , W V Represent the weights of Q, K, and V respectively;

[0023] Apply cross attention between Q, K and V and get the output query embedding z through the feed-forward network:

[0024] z=LayerNorm(selfAtt(Q,Q,Q))+CrossAtt(Input,K,V)+FFN(Input)

[0025] Among them, LayerNorm() is the layer normalization layer, selfAtt() is the self-attention mechanism layer, CrossAtt() is the cross attention module, FFN() is the forward feedback network layer, and Input is the input vision, point cloud, and command;

[0026] Align the image features embedded in the i-th output query using contrastive learning and point cloud features The goal of contrastive learning is to minimize the contrastive loss function

[0027] Contrastive loss function for

[0028]

[0029] Among them, K is the total number of output queries, Sim() is the similarity measurement function;

[0030] The output query embedding is mapped to the text modality through the language mapping layer to obtain the intent modality instruction.

[0031] The live working generation system based on multi-modal instructions of the present invention comprises:

[0032] The machine learning task model for dynamic live working environment is used to build a machine learning task model for dynamic live working environment based on machine learning according to the intention instructions in historical live working tasks and their corresponding robot action sequences, determine the distribution network live working sample library, and use it for fine-tuning training of the power industry model;

[0033] The operation prompt template is used to classify and describe the operation based on the historical distribution network live robot operation process and prompt learning to obtain the operation prompt template. The power industry model combines the input intention modal instruction with the operation prompt template to generate an atomic action sequence;

[0034] Based on the attention multimodal model, it is used to convert visual,point cloud,commands into intent modal commands;

[0035] The power industry model is used to receive intention modal instructions and generate atomic action sequences in combination with operation prompt templates.

[0036] Furthermore, the machine learning task model for the dynamic live working environment analyzes the voice instructions and input instructions in the historical working tasks according to the historical visual information, historical environmental information, historical working tasks and historical performance indicators, and uses machine learning to identify the operator's working actions and the corresponding atomic actions of the robot in the historical visual information, and combines the historical environmental information and historical performance indicators to construct a machine learning task model for the dynamic live working environment based on the working environment, intention instructions and atomic actions of the robot, and obtain a distribution network live working sample library.

[0037] Furthermore, the attention-based multimodal model includes a visual module, a point cloud module and a language module; the visual module extracts image features y from the image input vis , the point cloud module extracts point cloud features y from the point cloud point , the language module extracts language features y from the language input ins According to the characteristics of the live task and the distribution of the data, a thinking chain is constructed for the feature extractor, language projection layer and prefix language adapter, and fine-tuning training is performed to adjust the parameters of the attention-based multimodal model.

[0038] Furthermore, the visual, point cloud, and command are input into the attention multimodal model to obtain the intention modal command, which is specifically as follows: the point cloud feature y is fused through cross attention point , image feature y vis and language feature y ins :

[0039] Use three linear layers to generate Query-Key-Value pairs:

[0040] Q=y point ·WQ

[0041] K=y vis ·W K

[0042] V=y ins ·W V

[0043] Among them, Q, K, and V represent Query, Key, and Value respectively, and W Q , W K , W V Represent the weights of Q, K, and V respectively;

[0044] Apply cross attention between Q, K and V and get the output query embedding z through the feed-forward network:

[0045] z=LayerNorm(selfAtt(Q,Q,Q))+CrossAtt(Input,K,V)+FFN(Input)

[0046] Among them, LayerNorm() is the layer normalization layer, selfAtt() is the self-attention mechanism layer, CrossAtt() is the cross attention module, FFN() is the forward feedback network layer, and Input is the input vision, point cloud, and command;

[0047] Align the image features embedded in the i-th output query using contrastive learning and point cloud features The goal of contrastive learning is to minimize the contrastive loss function

[0048] Contrastive loss function for

[0049]

[0050] Among them, K is the total number of output queries, Sim() is the similarity measurement function;

[0051] The output query embedding is mapped to the text modality through the language mapping layer to obtain the intent modality instruction.

[0052] The computer device of the present invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the steps of the above method are implemented when the processor executes the computer program.

[0053] The computer-readable storage medium of the present invention stores a computer program thereon, and the computer program implements the steps of the above method when executed by a processor.

[0054] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: 1. Based on machine learning, the present invention constructs a machine learning task model for dynamic live working environments according to the operator's video and voice intention instructions and their corresponding robot action sequences, determines a high-quality distribution network live working sample library, fine-tunes the power industry model, and performs operation classification description based on prompt learning to obtain an operation prompt template; 2. The attention multimodal model of the present invention extracts multimodal features and maps them to the intention instruction modality through fine-tuning training of the thinking chain. The power industry model generates an atomic action sequence through the intention modal instructions combined with the operation prompt template, which solves the problems of difficulty and low efficiency in the robot's semantic understanding, and the unsmooth on-site operation process, resulting in low operating efficiency and long operating time. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flow chart of a method for generating atomic action sequences for live working according to the present invention;

[0056] Figure 2 Schematic diagram of the flow chart for generating the atomic action sequence of the lead wire. DETAILED DESCRIPTION

[0057] The present invention will be further described below in conjunction with the accompanying drawings.

[0058] The method for generating live working based on multimodal instructions of the present invention comprises the following steps:

[0059] (1) Based on historical visual information, historical environmental information, historical work tasks, and historical performance indicators, the voice commands and input commands in the historical work tasks are analyzed, and machine learning is used to identify the operator's work actions and the corresponding robot's atomic actions in the historical visual information. Combined with historical environmental information and historical performance indicators, a machine learning task model for dynamic live working environments is constructed based on the working environment, intentional commands, and the robot's atomic actions, and a distribution network live working sample library is obtained.

[0060] Collect real-time data obtained by the robot's sensors during historical operations, such as visual information, force feedback, electromagnetic field strength, etc., as well as operation logs and performance indicators during the operation. Through data preprocessing, feature extraction and pattern recognition, a detailed data set reflecting the operating environment and robot behavior is constructed. 360-degree panoramic image fusion technology is used to parse the video data during the robot's operation to obtain more comprehensive visual information. The gestures of the staff issuing instructions are used as input data, which are matched to machine instructions to instruct the robot to make corresponding actions and match the robot action sequence. Similarly, the sound data and input instructions are matched to the relevant robot action sequence with the machine instructions. To process the operation text data, it is necessary to segment a sentence into multiple words, then count the number of times the same phrase appears, and extract a series of action keywords that the robot arm will perform, such as "wiring", "grasping", "lifting", "stripping", etc.

[0061] Analyze the operation requirements, identify the key links and potential risk points in the operation, and clarify the functions and performance indicators that the robot should have in the operation, such as accuracy, response speed, adaptability and safety.

[0062] By combining multimodal data fusion with deep learning environment perception algorithms, panoramic video stitching is performed in combination with multi-source image data to provide operators with more comprehensive environmental information. Operator gestures and voice input are used to assist the robot in the actions to be performed, and input text instructions are used to ensure the robot's understanding of the actions to be performed. By performing machine learning task modeling on the robot's operations in a dynamic live working environment, a high-quality training sample library is constructed.

[0063] (2) According to the historical distribution network live robot operation process, based on prompt learning, the operation classification description is carried out to obtain the operation prompt template.

[0064] The instructions are grouped according to the verbs used in the instructions of the historical distribution network live robot operation process to form a prompt template. Prompt templates include "grasp target", "lift target", etc. The prompt template describes the definition of the specific task plan, the requirements for generating instructions and the corresponding action plan. It emphasizes that the target object of the generated action should be constrained in the object list x s In order to mitigate object errors that cause unexecutable plans. For the object lists used in the dataset generation hints, the basic labels of existing instances in the scene are directly used.

[0065] (3) Construct an attention-based multimodal model and perform thought chain fine-tuning training. Input vision, point cloud, and instructions into the attention multimodal model to obtain intention modal instructions.

[0066] The attention-based multimodal model includes a visual module, a point cloud module, and a language module. The visual module is used to extract features from image input, represented by E vis :x vis →y vis ; Point cloud module, used to extract features from point clouds, denoted as E point :x point →y point ; Language module, used to extract features from a given language input, denoted as E ins :x ins →y ins .

[0067] The thought chain can be linear or nonlinear, and is designed according to the complexity of the task and the correlation between the data. According to the characteristics of the live task and the distribution of the data, the thought chain is constructed for the feature extractor, language projection layer, and prefix language adapter, and fine-tuning training is performed to adjust the parameters of the attention-based multimodal model so that it can better adapt to specific tasks. Fine-tuning training only requires a small number of samples. By showing specific tasks and specific reasoning processes, the large model can learn generalizable knowledge and apply it to reasoning in new tasks.

[0068] The output query embedding is represented as z∈R N×D , where D is the dimension of the embedding. The size of z is significantly smaller than the size of the visual features. Contrastive learning is used to align the image and point cloud to unify the scene information. Specifically, if the point cloud and image belong to the same class, their embeddings are considered as positive pairs, otherwise they are considered as negative samples for contrastive learning. and denote point cloud embedding and image embedding respectively, where forms positive eigenpairs, and Contrastive learning aims to cluster positive pairs together while separating negative pairs as much as possible. The goal of contrastive learning is to minimize the contrast loss. for

[0069]

[0070] Where Sim() is a similarity measure (i.e., cosine similarity) function. By optimizing the above losses, the model can effectively unify the scene differences between point clouds and images.

[0071] In order to effectively fuse point clouds, images and instructions, cross attention is used to fuse y point ,y vis and ins First, three linear layers are used to generate Query-Key-Value pairs:

[0072] Q=ypoint ·W Q ,K=y vis ·W K ,V=y ins ·W V

[0073] Then, cross attention is applied between Q, K and V, and the expressiveness of the model is enhanced through a feed-forward network (FFN). The core calculation process is as follows:

[0074] z=LayerNorm(selfAtt(Q,Q,Q))+CrossAtt(Input,K,V)+FFN(Input)

[0075] Where Att() is a multi-head attention module.

[0076] The present invention adopts the thinking chain visual language fine-tuning method, combining video input with task description, detailed planning and structured gerund pairs to perform model reasoning prompts. Specifically, according to the action atomic sequence of the distribution network live robot and task instructions, such as action instructions, voice instructions and input instructions, the thinking chain is determined to simulate the robot's operation process in a specific environment, and the needs of human operators are fully considered. The generated prompt language covers multiple types such as requests, commands and queries to adapt to diverse interaction scenarios.

[0077] To avoid overfitting, the present invention also designs a set of instruction prompts with different expressions but the same meaning. In this process, the model's patch feature extractor, language projection layer, and prefix language adapter are fine-tuned to more effectively capture time series information. The output query embedding is mapped to the text modality through the language mapping layer. plan ∈R N×D′ , which has the same dimension D' as the text embedding of the large model of the power industry in the language modality, provides input for the large model of the power industry to generate atomic action sequences, thereby improving the accuracy and robustness of the model.

[0078] (6) Use the distribution network live operation sample library to fine-tune the power industry model, input the intention modal instructions into the power industry model, and combine them with the operation prompt template to generate an atomic action sequence.

[0079] like Figure 2 As shown in the figure, taking the "wire connection" robot task as an example, the wire connection task uses voice commands or gesture action commands as input based on the attention multimodal model to conduct conversational interaction, determine the thought chain, extract multimodal features from voice commands or gesture action commands and map them to the intention command modality. The power industry model combines the intention command map modality with the operation prompt template, uses the retrieval enhancement mechanism to search the robot knowledge base, and generates the atomic action sequence of the task.

[0080] For example, the wire connection operation will include the following action sequence: robot preparation-wire grabbing and identification-force-controlled wire grabbing-force-controlled wire lifting-wire threading identification-force-controlled threading-wire stripping and retracting-wire stripping identification-wire stripping-force-controlled wire stripping-wire stripping and withdrawal-preparation wiring-force-controlled wiring-robot reset-take the right wire clamp. According to the thinking connection reasoning method, the order relationship of these actions is given and their logic is explained. After obtaining the action sequence, the robot action series is mapped into an atomic action sequence. For example, "force-controlled threading" can be decomposed into several atomic actions: turntable movement, camera recognition, robot arm carrying tools to the front of the wire, force-controlled hole search is turned on, photoelectric sensor determines the wire entering the hole, and determines success.

[0081] The computer device of the present invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the steps of the above method are implemented when the processor executes the computer program.

[0082] The computer-readable storage medium of the present invention stores a computer program thereon, and the computer program implements the steps of the above method when executed by a processor.

[0083] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0084] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0085] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0086] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

Claims

1. A method for generating live working based on multimodal instructions, characterized in that: The following steps are involved: Based on machine learning, according to the intention instructions in historical live working tasks and their corresponding robot action sequences, a machine learning task model for dynamic live working environments is constructed to determine the distribution network live working sample library; According to the historical distribution network live robot operation process, based on prompt learning, the operation classification description is carried out to obtain the operation prompt template; Construct a multimodal model based on attention and conduct fine-tuning training on the thinking chain; Build a power industry model and use the distribution network live operation sample library to fine-tune the power industry model; Input vision, point cloud, and instructions to the attention multimodal model to obtain the intention modality instruction; Input the intention modal instructions into the power industry model, combine them with the operation prompt template, and generate atomic action sequences.

2. The method for generating atomic action sequences for live working based on multimodal instructions according to claim 1 is characterized in that: The steps for determining the distribution network live working sample library are as follows: According to historical visual information, historical environmental information, historical work tasks and historical performance indicators, the voice commands and input commands in the historical work tasks are analyzed, and machine learning is used to identify the operator's work actions and the corresponding robot's atomic actions in the historical visual information. Combined with historical environmental information and historical performance indicators, a machine learning task model for dynamic live working environments is constructed based on the working environment, intention commands and the robot's atomic actions to obtain a distribution network live working sample library.

3. The method for generating atomic action sequences for live working based on multimodal instructions according to claim 1 is characterized in that: The attention-based multimodal model includes a visual module, a point cloud module, and a language module; the visual module extracts image features y from the image input. vis , the point cloud module extracts point cloud features y from the point cloud point , the language module extracts language features y from the language input ins According to the characteristics of the live task and the distribution of the data, a thinking chain is constructed for the feature extractor, language projection layer and prefix language adapter, and fine-tuning training is performed to adjust the parameters of the attention-based multimodal model.

4. The method for generating atomic action sequences for live working based on multimodal instructions according to claim 3 is characterized in that: Input vision, point cloud, and command to the attention multimodal model to obtain the intention modal command, as follows: Fusion of point cloud features y by cross attention point , image feature y vis and language feature y ins : Use three linear layers to generate Query-Key-Value pairs: Q=y point ·W Q K=y vis ·W K V=y ins ·W V Among them, Q, K, and V represent Query, Key, and Value respectively, and W Q , W K , W V Represent the weights of Q, K, and V respectively; Apply cross attention between Q, K and V and get the output query embedding z through the feed-forward network: z=LayerNorm(selfAtt(Q,Q,Q))+CrossAtt(Input,K,V)+FFN(Input) Among them, LayerNorm() is the layer normalization layer, selfAtt() is the self-attention mechanism layer, CrossAtt() is the cross attention module, FFN() is the forward feedback network layer, and Input is the input vision, point cloud, and command; Align the image features embedded in the i-th output query using contrastive learning and point cloud features The goal of contrastive learning is to minimize the contrastive loss function Contrastive loss function for Among them, K is the total number of output queries, Sim() is the similarity measurement function; The output query embedding is mapped to the text modality through the language mapping layer to obtain the intent modality instruction.

5. A live working generation system based on multi-modal instructions, characterized in that: include: The machine learning task model for dynamic live working environment is used to build a machine learning task model for dynamic live working environment based on machine learning according to the intention instructions in historical live working tasks and their corresponding robot action sequences, determine the distribution network live working sample library, and use it for fine-tuning training of the power industry model; The operation prompt template is used to classify and describe the operation based on the historical distribution network live robot operation process and prompt learning to obtain the operation prompt template. The power industry model combines the input intention modal instruction with the operation prompt template to generate an atomic action sequence; Based on the attention multimodal model, it is used to convert visual,point cloud,commands into intent modal commands; The power industry model is used to receive intention modal instructions and generate atomic action sequences in combination with operation prompt templates.

6. The live working generation system based on multi-modal instructions according to claim 5, characterized in that: The machine learning task model for dynamic live working environments analyzes voice commands and input commands in historical working tasks based on historical visual information, historical environmental information, historical working tasks, and historical performance indicators, uses machine learning to identify the operator's working actions and the corresponding atomic actions of the robot in the historical visual information, and combines the historical environmental information and historical performance indicators to construct a machine learning task model for dynamic live working environments based on the working environment, intentional commands, and atomic actions of the robot, thereby obtaining a distribution network live working sample library.

7. The live working generation system based on multimodal instructions according to claim 5, characterized in that: The attention-based multimodal model includes a visual module, a point cloud module and a language module; the visual module extracts image features y from the image input vis , the point cloud module extracts point cloud features y from the point cloud point , the language module extracts language features y from the language input ins According to the characteristics of the live task and the distribution of the data, a thinking chain is constructed for the feature extractor, language projection layer and prefix language adapter, and fine-tuning training is performed to adjust the parameters of the attention-based multimodal model.

8. The live working generation system based on multimodal instructions according to claim 5, characterized in that: The attention multimodal model is fed with vision, point cloud, and instructions to obtain the intention modal instruction, specifically as follows: point cloud feature y is fused through cross attention point , image feature y vis and language feature y ins : Use three linear layers to generate Query-Key-Value pairs: Q=y point ·W Q K=y vis ·W K V=y ins ·W V Among them, Q, K, and V represent Query, Key, and Value respectively, and W Q , W K , W V Represent the weights of Q, K, and V respectively; Apply cross attention between Q, K and V and get the output query embedding z through the feed-forward network: z=LayerNorm(selfAtt(Q,Q,Q))+CrossAtt(Input,K,V)+FFN(Input) Among them, LayerNorm() is the layer normalization layer, selfAtt() is the self-attention mechanism layer, CrossAtt() is the cross attention module, FFN() is the forward feedback network layer, and Input is the input vision, point cloud, and command; Align the image features embedded in the i-th output query using contrastive learning and point cloud features The goal of contrastive learning is to minimize the contrastive loss function Contrastive loss function for Among them, K is the total number of output queries, Sim() is the similarity measurement function; The output query embedding is mapped to the text modality through the language mapping layer to obtain the intent modality instruction.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images