Generative instruction control robot method, device, equipment and medium
By obtaining video data and 2D trajectory sketches, and using the Transformer module to generate robot control instructions, it solves the problem that robots cannot independently form action instructions and tasks in the prior art, and realizes the robot's ability to independently perform complex tasks and quickly reason.
Patent Information
- Application Number
- CN202411321469.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2025-01-14
AI Technical Summary
Existing inspection robots based on multi-scale transformers cannot independently form action instructions, and the tasks are single, so they cannot judge and plan complex tasks, and lack the perception of surrounding space and terrain information.
通过获取机器人周围环境的视频数据和2D轨迹草图,使用Image tokenizer模块将图像与轨迹草图连接并进行token化,利用Transformer模块进行解码输出机器人执行任务的控制指令,采用模仿学习的方式训练动作策略,实现自主化指令生成。
The robot is autonomous, and can perform complex tasks without human control, reduce the complex robot logic control code, improve the model inference speed and efficiency, and realize the robot miniaturization and rapid inference at the edge.
Smart Images

Figure CN119310991A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of robot control technology, and in particular to a transformer-based generative instruction control robot method, device, equipment and medium. Background Art
[0002] With the major breakthroughs and progress in multiple subfields of machine learning (NLP / CV), transformers came into being. By making full use of large-scale and diverse data sets, a representation model that effectively absorbs all data is constructed. This technology uses existing and known knowledge to generate a new thing with certain expressive capabilities. Transformer has the following important parts and features:
[0003] 1. Self-Attention: This is one of the core concepts of Transformer, which enables the model to consider all positions in the input sequence at the same time, rather than processing them step by step like recurrent neural networks (RNNs) or convolutional neural networks (CNNs). The self-attention mechanism allows the model to give different attention weights to different parts of the input sequence, thereby better capturing semantic relationships.
[0004] 2. Multi-Head Attention: The self-attention mechanism in Transformer is extended to multiple attention heads, each of which can learn different attention weights to better capture different types of relationships. Multi-head attention allows the model to process different information subspaces in parallel.
[0005] 3. Stacked Layers: Transformer is usually composed of multiple identical encoder and decoder layers stacked together. These stacked layers help the model learn complex feature representations and semantics.
[0006] 4. Positional Encoding: Since Transformer has no built-in sequence position information, it requires additional positional encoding to express the position order of words in the input sequence.
[0007] 5. Residual Connections and Layer Normalization: These techniques help alleviate the vanishing and exploding gradient problems during training, making the model easier to train.
[0008] 6. Encoder and Decoder: Transformer usually consists of an encoder for processing the input sequence and a decoder for generating the output sequence, which makes it suitable for sequence-to-sequence tasks such as machine translation.
[0009] The prior art provides a patrol robot and patrol method based on a multi-scale transformer, which includes an image acquisition module, a computing module and a storage module. The image acquisition module includes two cameras with different focal lengths, the image acquisition module is used to acquire multi-scale images, the computing module uses a transformer model to process the acquired multi-scale images, and the storage module is used to store image data. At the same time, cameras with different focal lengths are used to shoot images of different scales, which solves the problem of too small targets and insufficient recognition accuracy caused by the wide-angle lens. The multi-scale transformer model is used to recognize the indicator light status and instrument characters. The accuracy of target recognition is further improved. In real life, it can greatly reduce the labor cost of patrol work and improve the efficiency and quality of patrol work.
[0010] Technical disadvantages of this method include:
[0011] 1. Because it does not have the function of processing natural language, the robot cannot automatically form action instructions and needs human control.
[0012] 2. The task content is very simple, just identifying the indicator light status and instrument characters, and it is impossible to make effective judgments and actions for complex tasks.
[0013] 3. Because it has no perception of the surrounding space or terrain information, it is unable to autonomously issue instructions for obstacle avoidance and path planning. Summary of the invention
[0014] In view of the above problems, the present invention provides a generative instruction control robot method, device, equipment and medium for overcoming the above problems or at least partially solving the above problems.
[0015] The present invention provides the following scheme:
[0016] A generative instruction control robot method, comprising:
[0017] Obtaining video data containing information about the robot's surrounding environment and a 2D trajectory sketch; the 2D trajectory sketch at least includes any trajectory that the robot needs to take to perform a target task;
[0018] Acquire a plurality of continuous sequence images from the video data, connect the plurality of continuous sequence images with the 2D trajectory sketch in the feature dimension through an Image tokenizer module, and tokenize the spliced images through a model pre-trained by ImageNet;
[0019] The required number of tokens is compressed through the Token learner module to obtain the target token combination;
[0020] The target token combination is decoded by a transformer module to output control instructions for the robot to perform the task, and the control instructions include an action strategy trained by imitation learning.
[0021] Preferably, the video data is collected by a camera of the robot or recorded and collected by an external camera.
[0022] Preferably, the 2D trajectory sketch is obtained by any of the following methods:
[0023] Manually draw the 2D trajectory sketch;
[0024] Generate the 2D trajectory sketch by estimating the robot pose trajectory from the video by watching the actual demonstration;
[0025] The 2D trajectory sketch is generated by VLM.
[0026] Preferably, generating the 2D trajectory sketch by VLM comprises:
[0027] By inputting a description, LLM generates a series of 3D poses and draws the 2D trajectory sketch on the initial image;
[0028] Alternatively, a text-guided image generation model is used to generate the 2D trajectory sketch.
[0029] Preferably: several 300*300 continuous sequence images are used as input, a feature map with a shape of 9*9*512 is output in the convolution layer, and the output feature map is flattened.
[0030] Preferably: compressing the required number of tokens through the Token learner module to obtain the target token combination includes:
[0031] Map low-value tokens to high-value tokens.
[0032] Preferably: the action strategy obtained by training in an imitation learning manner includes:
[0033] Use the Transformer parameterized strategy π to map the input i,{xj}tj=0i,{xj}j=0t to the sequence Action output a t Mapping to a sequence Learning the mapping using Transformer
[0034] Where: Input ξ h And the output y k Both represent a text token.
[0035] A generative instruction control robot device, characterized in that it is used to execute the above generative instruction control robot method, and the device comprises:
[0036] A data acquisition unit, used to acquire video data containing information about the robot's surrounding environment and a 2D trajectory sketch; the 2D trajectory sketch at least includes any trajectory that the robot needs to pass through to perform the target task;
[0037] A tokenization unit is used to obtain a plurality of continuous sequence images from the video data, connect the plurality of continuous sequence images with the 2D trajectory sketch in a feature dimension through an Imagetokenizer module, and tokenize the spliced images through a model pre-trained by ImageNet;
[0038] The token quantity compression unit is used to compress the required token quantity through the Token learner module to obtain the target token combination;
[0039] A control instruction output unit is used to decode the target token combination through a transformer module to output a control instruction for the robot to perform a task, wherein the control instruction includes an action strategy obtained by training through imitation learning.
[0040] A generative instruction controlled robot device, the device comprising a processor and a memory:
[0041] The memory is used to store program code and transmit the program code to the processor;
[0042] The processor is used to execute the above-mentioned generative instruction control robot method according to the instructions in the program code.
[0043] A computer-readable storage medium is used to store program codes, and the program codes are used to execute the above-mentioned generative instruction control robot method.
[0044] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0045] The embodiments of the present application provide a generative instruction control robot method, device, equipment and medium. By using TokenLearner to reduce the tokens generated by the pre-trained model, and only calculating these tokens once, and reusing them in the subsequent network layer for future reasoning, the model reasoning speed is increased to less than 100ms. Ensure that the robot can smoothly receive the generated instructions to perform subsequent tasks. The trained generative model builds an autonomous instruction generation model based on transformer, and there is no need to write complicated robot logic control code. The model training set constructs a training data set for a multimodal robot by collecting a large number of task videos and drawing trajectory sketches in three ways. The robot can be made autonomous and gradually separated from the manual issuance of instructions to control the robot. The robot can imitate human behavior to perform some complex tasks. The rapid reasoning of large generative models on robots and the miniaturization of complex network models at the edge are realized.
[0046] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0048] Figure 1 is a flow chart of a method for controlling a robot using generative instructions provided by an embodiment of the present invention;
[0049] Figure 2 is a system block diagram of a generative instruction control robot method provided by an embodiment of the present invention;
[0050] Figure 3 is a processing flow chart of an image word segmenter provided by an embodiment of the present invention;
[0051] Figure 4 is a processing flow chart of a token learner module provided in an embodiment of the present invention;
[0052] Figure 5 is a processing flow chart of a transformer module provided by an embodiment of the present invention;
[0053] Figure 6is a schematic diagram of a generative instruction controlled robot device provided by an embodiment of the present invention;
[0054] Figure 7 It is a schematic diagram of a generative instruction controlled robot device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0055] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.
[0056] See also Figure 1 , is a generative instruction control robot method provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include:
[0057] S101: Acquire video data containing information about the robot's surrounding environment and a 2D trajectory sketch; the 2D trajectory sketch includes at least one trajectory that the robot needs to take to perform a target task; in specific implementation, the embodiment of the present application can provide the video data to be collected by the robot's camera or recorded and collected using an external camera.
[0058] The 2D trajectory sketch is obtained by any of the following methods:
[0059] Manually draw the 2D trajectory sketch;
[0060] Generate the 2D trajectory sketch by estimating the robot pose trajectory from the video by watching the actual demonstration;
[0061] The 2D trajectory sketch is generated by VLM.
[0062] Further, generating the 2D trajectory sketch by VLM includes:
[0063] By inputting a description, LLM generates a series of 3D poses and draws the 2D trajectory sketch on the initial image;
[0064] Alternatively, a text-guided image generation model is used to generate the 2D trajectory sketch.
[0065] S102: Obtain a number of continuous sequence images from the video data, connect the several continuous sequence images with the 2D trajectory sketch in the feature dimension through the Image tokenizer module, and tokenize the spliced images through the imagenet pre-trained model; in specific implementation, the embodiment of the present application can provide a number of 300*300 continuous sequence images as input, output a feature map with a shape of 9*9*512 in the convolution layer, and flatten the output feature map.
[0066] S103: The target token combination is obtained by compressing the required number of tokens through the Token learner module; in specific implementation, the embodiment of the present application can provide mapping of low-value tokens to high-value tokens.
[0067] S104: Decode the target token combination through the transformer module to output the control instructions for the robot to perform the task, and the control instructions include the action strategy trained by imitation learning.
[0068] The action strategies trained by imitation learning include:
[0069] Use the Transformer parameterized strategy π to map the input i,{xj}tj=0i,{xj}j=0t to the sequence Action output a t Mapping to a sequence Learning the mapping using Transformer
[0070] Where: Input ξ h And the output y k Both represent a text token.
[0071] The generative command control robot method provided in the embodiment of the present application realizes autonomous object recognition of robots such as robot dogs based on transformers, directly generates task instructions in real time from the model, controls the robot to perform the next action, and is free from human control and cumbersome and complicated code logic. The complex tasks are decomposed, so that the robot can autonomously perform multiple tasks in areas that humans cannot reach, obtain information and provide feedback.
[0072] The following describes in detail the method provided by the embodiment of the present application by taking the robot dog for example to generate instructions. Figure 2 .
[0073] 1. Input data preparation
[0074] The input data of the method provided in the embodiment of the present application is divided into two parts, video data and 2D trajectory sketch.
[0075] Video data can be collected directly by the robot's camera or recorded and collected using an external camera. The video data is mainly used to obtain the environmental information of the space where the robot and the task point are located, so that when the control instruction is subsequently planned, the objects between the robot's current position and the task point where the task is to be executed can be accurately identified based on the environmental information, thereby determining whether obstacles need to be avoided based on each object.
[0076] 2D trajectory sketches can be created in a variety of ways:
[0077] 1. Manually hand-drawn 2D trajectory sketch. This method is simple and fast, and does not require actual demonstration again.
[0078] 2. By watching the actual demonstration, estimate the trajectory of the robot posture from the video and use it to generate a 2D trajectory sketch.
[0079] 3. Generate through VLM. One is to input a description, LLM writes and generates a series of 3D poses, and can draw a 2D trajectory sketch on the initial image. The other is to use the text-guided image generation model to generate a task 2D trajectory sketch, providing the initial image and language instructions.
[0080] The 2D trajectory sketch provided in the embodiment of the present application is mainly used to provide the robot with any ideal trajectory that needs to be taken to perform the current task. The ideal trajectory appears in the form of 2D, which may include the current position point of the robot and the task point that needs to be reached. For example, a straight line or an arbitrary curve from the current position point to the task point can be drawn as a 2D trajectory sketch.
[0081] Since the sketch is randomly drawn without considering the possible obstacles between the robot's current position and the task point, the robot needs to generate a control instruction through video data combined with the 2D trajectory sketch. The control instruction can include the target travel trajectory obtained after optimization, as well as the corresponding actions required to cross obstacles during the action. It can also include the desired actions required to be performed at the end after reaching the task point, etc.
[0082] The robot action strategy is trained using imitation learning, using Transformer to parameterize the strategy π and map the input i,{xj}tj=0i,{xj}j=0t to the sequence Action output a t Mapping to a sequence Learning the mapping using Transformer
[0083] Where: Input ξ h And the output y k Both represent a text token.
[0084] 2. Image tokenizer module (image tokenizer), see Figure 3 .
[0085] First, the video data captures 6 consecutive images, and then connects the 6 consecutive images with the trajectory sketch in the feature dimension; the images are tokenized using the model pre-trained by ImageNet. The connection refers to connecting each of the 6 consecutive images with the corresponding 6 trajectory points on the 2D trajectory sketch.
[0086] In the specific implementation, a continuous sequence of 6 300*300 images is used as input, and the feature map with a shape of 9*9*512 is output in the last convolution layer, and the output feature map is flattened. For the additional input channels of the word segmenter, the new weights of the first convolution layer are all initialized to zero.
[0087] 3. Token learner module, see Figure 4 .
[0088] Since the robot dog has limited computing power, in order to improve the reasoning speed, the number of tokens required is compressed. Therefore, the token learner is introduced. This is an element-by-element attention module that maps a large number of tokens to better tokens. Image tokens are selected based on the information, and important token combinations are passed to the following transformer layer. This module ultimately needs to sample all visual tags corresponding to each image to only 8 final tags.
[0089] 4. Transformer module, see Figure 5 .
[0090] The six consecutive historical images are connected to form 48 tokens and input into the transformer backbone. After the transformer decoder, the control instructions for the robot to perform the task are output.
[0091] In summary, the generative instruction control robot method provided in this application reduces the tokens generated by the pre-trained model by using TokenLearner, calculates these tokens only once, and reuses them in subsequent network layers for future reasoning, thereby increasing the model reasoning speed to less than 100ms. Ensure that the robot can smoothly receive the generated instructions to perform subsequent tasks. The trained generative model builds an autonomous instruction generation model based on transformer, and there is no need to write complicated robot logic control code. The model training set constructs a training data set for a multimodal robot by collecting a large number of task videos and drawing trajectory sketches in three ways. The robot can be made autonomous and gradually separated from the manual issuance of instructions to control the robot. The robot can imitate human behavior to perform some complex tasks. The rapid reasoning of large generative models on robots and the miniaturization of complex network models at the edge are realized.
[0092] See also Figure 6 , the embodiment of the present application can also provide a generative instruction control robot device, such as Figure 6 As shown, the device for executing the above-mentioned generative instruction control robot method may include:
[0093] The data acquisition unit 601 is used to acquire video data containing information about the robot's surrounding environment and a 2D trajectory sketch; the 2D trajectory sketch at least includes any trajectory that the robot needs to pass through to perform the target task;
[0094] A tokenization unit 602 is used to obtain a plurality of continuous sequence images from the video data, connect the plurality of continuous sequence images with the 2D trajectory sketch in the feature dimension through an Imagetokenizer module, and tokenize the spliced images through a model pre-trained by ImageNet;
[0095] A token quantity compression unit 603 is used to compress the required token quantity through the Token learner module to obtain a target token combination;
[0096] The control instruction output unit 604 is used to decode the target token combination through the transformer module and output the control instruction for the robot to perform the task, and the control instruction includes the action strategy obtained by training through imitation learning.
[0097] The embodiment of the present application may also provide a generative instruction controlled robot device, the device comprising a processor and a memory:
[0098] The memory is used to store program code and transmit the program code to the processor;
[0099] The processor is used to execute the steps of the above-mentioned generative instruction control robot method according to the instructions in the program code.
[0100] like Figure 7 As shown, a generative instruction controlled robot device provided in an embodiment of the present application may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through the communication bus 13.
[0101] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.
[0102] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in an embodiment of a method for controlling a robot using generative instructions.
[0103] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:
[0104] Obtaining video data containing information about the robot's surrounding environment and a 2D trajectory sketch; the 2D trajectory sketch at least includes any trajectory that the robot needs to take to perform a target task;
[0105] A plurality of continuous sequence images are obtained from the video data, and the plurality of continuous sequence images are connected with the 2D trajectory sketch in the feature dimension through an Image tokenizer module, and the spliced images are tokenized through a model pre-trained by ImageNet;
[0106] The required number of tokens is compressed through the Token learner module to obtain the target token combination;
[0107] The target token combination is decoded by a transformer module to output control instructions for the robot to perform the task, and the control instructions include an action strategy trained by imitation learning.
[0108] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required for at least one function (such as a file creation function, a data reading and writing function), etc.; the data storage area can store data created during use, such as initialization data, etc.
[0109] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.
[0110] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.
[0111] Of course, it should be noted that Figure 7 The structure shown does not constitute a limitation on the generative instruction control robot device in the embodiment of the present application. In actual applications, the generative instruction control robot device may include Figure 7 More or fewer components than shown, or combinations of certain components.
[0112] The embodiment of the present application may also provide a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the steps of the above-mentioned generative instruction control robot method.
[0113] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0114] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.
[0115] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.
[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims
1. A generative instruction control robot method, characterized in that: include: Obtaining video data containing information about the robot's surrounding environment and a 2D trajectory sketch; the 2D trajectory sketch at least includes any trajectory that the robot needs to take to perform a target task; Acquire a plurality of continuous sequence images from the video data, connect the plurality of continuous sequence images with the 2D trajectory sketch in the feature dimension through an Image tokenizer module, and tokenize the spliced images through a model pre-trained by ImageNet; The required number of tokens is compressed through the Tokenlearner module to obtain the target token combination; The target token combination is decoded by a transformer module to output control instructions for the robot to perform the task, and the control instructions include an action strategy trained by imitation learning.
2. The method for controlling a robot by generative instructions according to claim 1, characterized in that: The video data is collected by the robot's camera or recorded by an external camera.
3. The method for controlling a robot by generative instructions according to claim 1, characterized in that: The 2D trajectory sketch is obtained by any of the following methods: Manually draw the 2D trajectory sketch; Generate the 2D trajectory sketch by estimating the trajectory of the robot posture from the video by watching the actual demonstration; The 2D trajectory sketch is generated by VLM.
4. The method for controlling a robot by generative instructions according to claim 3, characterized in that: Generating the 2D trajectory sketch by VLM includes: By inputting a description, LLM generates a series of 3D poses and draws the 2D trajectory sketch on the initial image; Alternatively, a text-guided image generation model is used to generate the 2D trajectory sketch.
5. The method for controlling a robot by generative instructions according to claim 1, characterized in that: Take several 300*300 continuous sequence images as input, output a feature map with a shape of 9*9*512 in the convolution layer, and flatten the output feature map.
6. The method for controlling a robot by generative instructions according to claim 1, characterized in that: The target token combination obtained by compressing the required number of tokens through the Tokenlearner module includes: Map low-value tokens to high-value tokens.
7. The method for controlling a robot by generative instructions according to claim 1, characterized in that: The action strategies trained by imitation learning include: Use the Transformer parameterized strategy π to map the input i,{xj}tj=0i,{xj}j=0t to the sequence Action output a t Mapping to a sequence Learning the mapping using Transformer Where: Input ξ h And the output y k Both represent a text token.
8. A generative instruction controlled robot device, characterized in that: The device is used to execute the generative finger control robot method according to any one of claims 1 to 7, comprising: A data acquisition unit, used to acquire video data containing information about the robot's surrounding environment and a 2D trajectory sketch; the 2D trajectory sketch at least includes any trajectory that the robot needs to pass through to perform the target task; A tokenization unit is used to obtain a plurality of continuous sequence images from the video data, connect the plurality of continuous sequence images with the 2D trajectory sketch in a feature dimension through an Imagetokenizer module, and tokenize the spliced images through a model pre-trained by ImageNet; The token quantity compression unit is used to compress the required number of tokens through the Tokenlearner module to obtain the target token combination; A control instruction output unit is used to decode the target token combination through a transformer module to output a control instruction for the robot to perform a task, wherein the control instruction includes an action strategy obtained by training through imitation learning.
9. A generative instruction controlled robot device, characterized in that: The device comprises a processor and a memory: The memory is used to store program codes and transmit the program codes to the processor; The processor is used to execute the generative instruction control robot method according to any one of claims 1 to 7 according to the instructions in the program code.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the generative instruction control robot method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Robot control method and device based on visual language pre-training model and medium
CN115933387A