Method and apparatus for controlling robot, and medium and electronic device

Abstract

The present disclosure belongs to the technical field of robot control, and relates to a method and apparatus for controlling a robot, and a medium and an electronic device. The method comprises: acquiring a target picture, which is collected by a camera apparatus that is mounted on a robot, and acquiring a target instruction for controlling the robot; inputting the target picture into a trained visual model, such that the trained visual model outputs a visual embedding vector; acquiring a text embedding vector corresponding to the target instruction, and inputting the visual embedding vector and the text embedding vector into a trained language model, such that the trained language model outputs a target numerical value; and decoding the target numerical value, so as to obtain a movement parameter, wherein the movement parameter is used for controlling the movement of the robot. By means of the present disclosure, a robot is driven to perform various motions to complete an instruction of a user, such that human-robot interaction is more intelligent and freer, thereby meeting usage requirements in various application scenarios, increasing the success rate of specific task execution, and enlarging a usage range of the robot.

Description

Method, device, medium and electronic equipment for controlling robot

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese patent application number 202410276322.2, filed with the Patent Office of China on March 11, 2024, entitled “Method, device, medium and electronic device for controlling robots,” the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0003] The present disclosure belongs to the technical field of robot control, and in particular, relates to a method for controlling a robot, an apparatus for controlling a robot, a non-transitory computer-readable storage medium, and an electronic device. Background Art

[0004] The multimodal macromodel of embodied intelligence refers to an intelligent system that perceives and acts based on the physical body. It acquires information through interaction with the environment, understands problems and makes decisions through the macromodel, and transmits these actions to robots, thereby generating intelligent behavior and adaptability. The rapid development of multimodal macromodels and physical simulation environments, along with the continuous improvement of perception, cognition, reasoning, decision-making, and autonomous learning capabilities, has significantly promoted the promotion and use of embodied intelligence.

[0005] Currently, most embodied intelligence models are pipeline models, integrating an object recognition model, an intent slot model, and a robot action model to complete user commands. Pipeline models are subject to cascading errors, requiring each sub-model to achieve high accuracy to complete user tasks. Therefore, object categories and task instructions must be defined in advance, and the model must be trained after data is collated. This approach can meet business needs, but each new task must go through all the necessary steps, resulting in high development and maintenance costs. Furthermore, overall, accuracy and flexibility are limited. Summary of the Invention

[0006] To overcome the problems existing in the related art, the present disclosure provides a method for controlling a robot, an apparatus for controlling a robot, a non-transitory computer-readable storage medium, and an electronic device.

[0007] According to a first aspect of an embodiment of the present disclosure, there is provided a method for controlling a robot, the method comprising:

[0008] Obtaining a target image captured by a camera mounted on the robot, and obtaining a target instruction for controlling the robot;

[0009] Inputting the target image into a trained visual model so that the trained visual model outputs a visual embedding vector;

[0010] Obtaining a text embedding vector corresponding to the target instruction, and inputting the visual embedding vector and the text embedding vector into a trained language model so that the trained language model outputs a target value;

[0011] The target value is decoded to obtain motion parameters, and the motion parameters are used to control the motion of the robot.

[0012] Optionally, the target picture includes: a current picture captured by the shooting device at the current moment, and historical pictures captured by the shooting device before the current moment.

[0013] Optionally, before inputting the target image into the trained visual model, the method further includes:

[0014] Acquire first data and a first value corresponding to the first data, and acquire a first text vector corresponding to the first text data, where the first data includes first image data and first text data;

[0015] The first image data is input into the visual model to be trained, and the first text vector is input into the language model to be trained to obtain a first value, so that the training of the visual model to be trained is completed to obtain a trained visual model.

[0016] Optionally, inputting the target image into a trained visual model so that the trained visual model outputs a visual embedding vector includes:

[0017] Inputting the target image into a trained visual model so that the trained visual model outputs a first image vector;

[0018] The first image vector is mapped through a linear layer to obtain a visual embedding vector.

[0019] Optionally, obtaining the text embedding vector corresponding to the target instruction includes:

[0020] Performing text vectorization on the target instruction to obtain a second text vector;

[0021] The second text vector is mapped by a text embedding layer to obtain a text embedding vector.

[0022] Optionally, before inputting the visual embedding vector and the text embedding vector into the trained language model, the method further comprises:

[0023] Acquire second image data and a second value corresponding to the second image data;

[0024] The second image data is input into the language model to be trained, so that the language model to be trained outputs the second value, so as to fine-tune the language model to be trained to obtain a trained language model.

[0025] Optionally, the second picture data includes third picture data and fourth picture data, and the second value includes a third value and a fourth value.

[0026] Inputting the second image data into the language model to be trained, causing the language model to be trained to output the second value, and fine-tuning the language model to be trained to obtain a trained language model, includes:

[0027] Inputting the third image data into a language model to be trained, so that the language model to be trained outputs the third value, and performing initial fine-tuning on the language model to be trained to obtain an initially fine-tuned language model;

[0028] The fourth image data is input into the language model after initial fine-tuning, so that the language model after initial fine-tuning outputs a fourth value, and the language model after initial fine-tuning is advanced fine-tuned to obtain a trained language model.

[0029] Optionally, before inputting the third image data into the language model to be trained, the method further includes:

[0030] Performing data cleaning on the third value to obtain a cleaned third value.

[0031] Optionally, the third value and the fourth value adopt a standard representation of a reinforcement learning dataset.

[0032] Optionally, performing data cleaning on the third value to obtain a cleaned third value includes:

[0033] Unifying the value range of the third value according to the specification of the reinforcement learning dataset to obtain a unified third value; and / or

[0034] Unifying the representation of the third value according to the specification of the reinforcement learning dataset to obtain a unified third value; and / or

[0035] The third value is eliminated according to the specification of the reinforcement learning data set to obtain the eliminated third value.

[0036] Optionally, the target values ​​include: a first group of values, a second group of values ​​and a third group of values, and the motion parameters include: position parameters, rotation parameters and state parameters, the first group of values ​​represents the position parameters, the second group of values ​​represents the rotation parameters, and the third group of values ​​represents the state parameters.

[0037] Optionally, decoding the target value to obtain the motion parameter includes:

[0038] Obtaining a first preset value, a second preset value, a first threshold value, and a second threshold value corresponding to the first set of values, and calculating the first set of values ​​and the first preset value to obtain a first calculation result;

[0039] The first calculation result and the second preset value are calculated to obtain a second calculation result, and the second calculation result, the first threshold value, and the second threshold value are calculated to obtain a position parameter.

[0040] Optionally, decoding the target value to obtain the motion parameter includes:

[0041] Obtaining a first preset value, a second preset value, a third threshold value, and a fourth threshold value corresponding to the second set of values, and calculating the second set of values ​​and the first preset value to obtain a third calculation result;

[0042] The third calculation result and the second preset value are calculated to obtain a fourth calculation result, and the fourth calculation result, the third threshold value, and the fourth threshold value are calculated to obtain a rotation parameter.

[0043] Optionally, decoding the target value to obtain the motion parameter includes:

[0044] Obtaining a third preset value corresponding to the third set of values, and calculating the third set of values ​​and the third preset value to obtain a fifth calculation result;

[0045] The state parameter is determined according to the fifth calculation result.

[0046] According to a second aspect of an embodiment of the present disclosure, there is provided a device for controlling a robot, comprising:

[0047] a data acquisition module configured to acquire a target image captured by a camera mounted on the robot and to acquire a target instruction for controlling the robot;

[0048] a visual processing module configured to input the target image into a trained visual model so that the trained visual model outputs a visual embedding vector;

[0049] a language processing module configured to obtain a text embedding vector corresponding to the target instruction, and input the visual embedding vector and the text embedding vector into a trained language model so that the trained language model outputs a target value;

[0050] The decoding processing module is configured to decode the target value to obtain motion parameters, and the motion parameters are used to control the motion of the robot.

[0051] According to a third aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of any one of the methods described in the first aspect of the present disclosure are implemented.

[0052] According to a fourth aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0053] processor;

[0054] a memory for storing processor-executable instructions;

[0055] The processor is configured to: execute the executable instructions to implement the steps of any one of the methods described in the first aspect of the present disclosure.

[0056] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0057] In the method and apparatus provided by the exemplary embodiments of the present disclosure, target images and target instructions are input into the visual model and language model, respectively, for automatic task recognition. Based on the output target values, the robot is driven to perform various actions to complete the user's instructions, making human-computer interaction more intelligent and free, meeting the needs and uses of various application scenarios, improving the success rate of specific task execution, expanding the scope of robot use, and optimizing the user experience. Furthermore, through the application of an end-to-end multimodal large model, the embodied intelligence model no longer relies on the pipeline model, greatly improving the accuracy, flexibility, and practicality of the embodied intelligence system, and further reducing the development cost, maintenance cost, and usage cost of the embodied intelligence system. This is of great significance for promoting the development and application of embodied intelligence systems.

[0058] Other features and advantages of the present disclosure will be described in detail in the following detailed description.

[0059] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following detailed description, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0061] FIG1 schematically shows a flow chart of a method for controlling a robot in an exemplary embodiment of the present disclosure;

[0062] FIG2 schematically shows a flow chart of a method for training a visual model in an exemplary embodiment of the present disclosure;

[0063] FIG3 schematically shows a flow chart of a method for outputting a visual embedding vector in an exemplary embodiment of the present disclosure;

[0064] FIG4 schematically shows a flow chart of a method for obtaining a text embedding vector in an exemplary embodiment of the present disclosure;

[0065] FIG5 schematically shows a flow chart of a method for training a language model in an exemplary embodiment of the present disclosure;

[0066] FIG6 schematically shows a flow chart of a method for further training a language model in an exemplary embodiment of the present disclosure;

[0067] FIG7 schematically shows a flow chart of a method for data cleaning processing in an exemplary embodiment of the present disclosure;

[0068] FIG8 schematically shows a flow chart of a method for decoding a target value in an exemplary embodiment of the present disclosure;

[0069] FIG9 schematically shows a flow chart of another method for decoding a target value in an exemplary embodiment of the present disclosure;

[0070] FIG10 schematically shows a flow chart of another method for decoding a target value in an exemplary embodiment of the present disclosure;

[0071] FIG11 schematically shows a flowchart of a method for controlling a robot in an application scenario in an exemplary embodiment of the present disclosure;

[0072] FIG12 schematically shows a flow chart of a method for training a language model in an application scenario in an exemplary embodiment of the present disclosure;

[0073] FIG13 schematically shows a data set diagram of a manipulator robot in an application scenario in an exemplary embodiment of the present disclosure;

[0074] FIG14 schematically shows an interface diagram for collecting task data in an application scenario in an exemplary embodiment of the present disclosure;

[0075] FIG15 schematically shows an interface diagram using Google's RLDS specification representation in an application scenario in an exemplary embodiment of the present disclosure;

[0076] FIG16 schematically shows a schematic structural diagram of a device for controlling a robot in an exemplary embodiment of the present disclosure;

[0077] FIG17 schematically illustrates an electronic device for implementing a method for controlling a robot in an exemplary embodiment of the present disclosure;

[0078] FIG18 schematically illustrates another electronic device for implementing a method for controlling a robot in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0079] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0080] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the corresponding data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0081] The present disclosure provides a method for controlling a robot. FIG1 is a flow chart of a method for controlling a robot according to an exemplary embodiment. As shown in FIG1 , the method may include at least the following steps:

[0082] Step S110: Obtain a target image captured by a shooting device installed on the robot, and obtain a target instruction for controlling the robot.

[0083] Step S120: Input the target image into the trained visual model so that the trained visual model outputs a visual embedding vector.

[0084] Step S130. Obtain a text embedding vector corresponding to the target instruction, and input the visual embedding vector and the text embedding vector into the trained language model so that the trained language model outputs the target value.

[0085] Step S140: Decode the target value to obtain motion parameters, which are used to control the movement of the robot.

[0086] In the exemplary embodiments of the present disclosure, target images and target instructions are input into the visual model and language model, respectively, for automatic task recognition. Based on the output target values, the robot is driven to perform various actions to complete the user's instructions, making human-computer interaction more intelligent and free, meeting the needs and uses of various application scenarios, improving the success rate of specific task execution, expanding the scope of robot use, and optimizing the user experience. Furthermore, through the application of an end-to-end multimodal large model, the embodied intelligence model no longer relies on the pipeline model, greatly improving the accuracy, flexibility, and practicality of the embodied intelligence system, and further reducing the development cost, maintenance cost, and usage cost of the embodied intelligence system. This is of great significance for promoting the development and application of embodied intelligence systems.

[0087] The following describes in detail the various steps of the method for controlling the robot.

[0088] In step S110 , a target image captured by a shooting device installed on the robot is obtained, and a target instruction for controlling the robot is obtained.

[0089] In the exemplary embodiments of the present disclosure, the photographing device may be a camera or other devices, which is not particularly limited in the present exemplary embodiments.

[0090] When the robot is a robotic arm or the robot has a robotic arm device, the shooting device can be installed at any position of the robot, such as a wrist and a joint, etc., and this exemplary embodiment does not impose any special limitation on this.

[0091] In an optional embodiment, the target picture includes: a current picture captured by the photographing device at the current moment, and historical pictures captured by the photographing device before the current moment.

[0092] The target image may include images taken at the current moment by cameras at different positions of the robot as the current image, and may also include N historical images taken before the current moment, where N may be equal to 2 or 3, etc., or other values ​​may be set according to actual conditions. This exemplary embodiment does not impose any special limitations on this.

[0093] The target instruction may be an instruction issued by a user to control the action of the robot, such as "pick up the apple on the table".

[0094] In step S120 , the target image is input into the trained visual model so that the trained visual model outputs a visual embedding vector.

[0095] In an exemplary embodiment of the present disclosure, after a target image is acquired, the target image may be input into a trained visual model.

[0096] In an optional embodiment, Figure 2 shows a flow chart of a method for training a visual model. As shown in Figure 2, the method may include at least the following steps: in step S210, obtain first data and a first numerical value corresponding to the first data, and obtain a first text vector corresponding to the first text data, wherein the first data includes first image data and first text data.

[0097] The first data may be multimodal data, and may be obtained by collecting open-source image and text datasets. For example, the collected dataset may be a VQA (Visual Question Answering) dataset.

[0098] The images in the VQA dataset primarily consist of two parts: realistic images and abstract cartoon images. VQA-real contains 123,287 training images and 81,434 test images, primarily from the MS-COCO (large image dataset) dataset. Unlike some previous datasets, VQA-real includes binary questions. This dataset allows for multiple-choice questions, providing 17 additional incorrect answer candidates for each question. In total, VQA-real contains 614,163 questions, each with 10 answers from different viewers.

[0099] Specifically, the open source image dataset may be the first image data, and the open source text dataset may be the first text data.

[0100] The first numerical value corresponding to the first data may be a numerical value used to represent a word in the first text data in a vocabulary of a language model.

[0101] The first text vector may be obtained after performing text vectorization and mapping processing on the first text data.

[0102] In step S220, the first image data is input into the visual model to be trained, and the first text vector is input into the language model to be trained to obtain a first value, so that the training of the visual model to be trained is completed to obtain a trained visual model.

[0103] After obtaining the first image data and the first text vector, the first image data can be input into the visual model to be trained, and the first text vector can be input into the language model to be trained, so that the language model outputs a first numerical value, thereby completing the training of the visual model to be trained and obtaining a trained visual model.

[0104] In an optional embodiment, Figure 3 shows a flow chart of a method for outputting a visual embedding vector. As shown in Figure 3, the method may at least include the following steps: in step S310, the target image is input into the trained visual model so that the trained visual model outputs a first image vector.

[0105] The visual model may be a ViT (Vision Transformer, an image classification model) model or other models, which is not particularly limited in this exemplary embodiment.

[0106] The Transformer (an attention-based sequence model) was originally used primarily in the field of Natural Language Processing (NLP). In 2020, a paper (An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale) brought the Transformer from NLP to the field of Computer Vision (CV), achieving success in multiple visual tasks.

[0107] The main steps of the ViT model include the following:

[0108] Patch Embedding (an embedding method used in natural language processing tasks): First, the original input image is sliced.

[0109] Assume the input image size is 224×224. Cut the image into fixed-size 16×16 blocks, with each block being a patch. Therefore, the number of patches in each image is (224×224) / (16×16) = 196. After cutting, 196 patches of [16,16,3] are obtained. These patches can be fed into the Linear Projection of Flattened Patches (Embedding layer). This layer flattens the input sequence, resulting in 196 tokens in the output. Each token has a flattened dimension of 16×16×3 = 768, resulting in an output dimension of [196,768]. It's easy to see that Patch Embedding transforms a CV problem into an NLP problem through cutting and flattening.

[0110] Position Embedding (embedding position information into word embedding vector): Each patch of the image, like the text, has a sequence and cannot be arbitrarily disrupted, so it is necessary to add position information to each token.

[0111] Similar to the BERT (Bidirectional Encoder Representation from Transformers) model, a special character class token needs to be added. Therefore, the final sequence dimension to be input to the Transformer Encoder is [197,768]. Position Embedding is used to add position information.

[0112] Transformer Encoder: Input the sequence of dimension [197,768] into the standard Transformer Encoder.

[0113] MLP (Multilayer Perceptron) Head: The output of the Transformer Encoder is actually a sequence, but in the ViT model, only the class token output is used, which is fed into the MLP module to output the classification result. The MLP Head is used for the final classification.

[0114] Therefore, after the target image is input into the trained visual model, the trained visual model can be used to perform image encoding on the target image, thereby outputting a first image vector.

[0115] In step S320, the first image vector is mapped through a linear layer to obtain a visual embedding vector.

[0116] After outputting the first image vector, a linear layer may be used to map the first image vector to obtain a vision embedding vector.

[0117] It is worth noting that the size of the visual embedding vector and the subsequent text embedding vector should match to adapt to the requirements of the language model.

[0118] In step S130, a text embedding vector corresponding to the target instruction is obtained, and the visual embedding vector and the text embedding vector are input into the trained language model so that the trained language model outputs the target value.

[0119] In an exemplary embodiment of the present disclosure, a text embedding vector corresponding to the target instruction may also be obtained.

[0120] In an optional embodiment, FIG4 shows a flow chart of a method for obtaining a text embedding vector. As shown in FIG4 , the method may at least include the following steps: in step S410 , text vectorization is performed on the target instruction to obtain a second text vector.

[0121] Specifically, tokenization is the process of dividing input and output text into smaller units, which are processed by LLM (Large Language Model) AI (Artificial Intelligence) models.

[0122] Tokenization can help models handle different languages, vocabularies, and formats, and reduce computational and memory costs. It can also affect the quality and diversity of generated text by influencing the meaning and context of tokens.

[0123] Depending on the complexity and variability of the text, different approaches can be used for tokenization, such as rule-based, statistical, or neural methods.

[0124] Tokenization involves segmenting text into meaningful units to capture its semantic and syntactic structure. Various tokenization techniques can be used, such as word-level, subword-level, or character-level. Each technique has its own advantages and trade-offs, depending on the needs of a specific language and a specific task.

[0125] Byte Pair Encoding: Build subword vocabularies for AI models to combine frequently occurring character / subword pairs.

[0126] Subword-level tokenization: Splitting words into smaller units is important for complex languages ​​and vocabularies.

[0127] Word-level tokenization: Basic text tokenization for language processing. Each word is used as a different token, which is simple but limited.

[0128] Sentence fragments: Segment text using learned subword fragments, based on segmentation of learned subword fragments.

[0129] Tokenization: Subword units using different merging methods.

[0130] Byte-level tokenization: Use byte-level tokenization to handle text diversity, treating each byte as a token, which is very important for multilingual tasks.

[0131] Hybrid tokenization: Balancing fine details and interpretability, combining word-level and subword-level tokenization.

[0132] LLM has expanded its capabilities to handle multilingual and multimodal input. To accommodate the diversity of this data, specialized tokenization methods have been developed. By leveraging language-specific tokens or subwords, multilingual tagging handles multiple languages ​​within a single model. Multimodal tagging combines text with other modalities (e.g., images or audio), using techniques such as fusion or concatenation to efficiently represent different data sources.

[0133] Tokenization plays a crucial role in the efficiency, flexibility, and generalization capabilities of LLMs. By breaking text into smaller, manageable units, LLMs can process and generate text more efficiently, reducing computational complexity and memory requirements.

[0134] Furthermore, tokenization provides flexibility by adapting to different languages, domain-specific terminology, and even emerging text forms (such as internet slang or emoticons). This flexibility allows LLMs to handle a wide range of text inputs, enhancing their applicability in different domains and user contexts.

[0135] The choice of tokenization technique involves a trade-off between granularity and semantic understanding. Word-level tokenization captures the meaning of individual words but may encounter out-of-vocabulary terms or morphologically rich languages. Subword-level tokenization provides greater flexibility and handles OOV (out of vocabulary) terms by breaking words into subword units. However, correctly understanding the meaning of subword tokens in the context of the entire sentence is a challenge. The choice of tokenization technique depends on the specific task, language characteristics, and available computational resources.

[0136] Therefore, the text vectorization processing of the target instruction can be achieved through the tokenization technology to obtain the second text vector.

[0137] In step S420, the second text vector is mapped by a text embedding layer to obtain a text embedding vector.

[0138] Furthermore, the second text vector is mapped through the Text Embedding layer to obtain a text embedding vector.

[0139] In an optional embodiment, FIG5 shows a flow chart of a method for training a language model. As shown in FIG5 , the method may include at least the following steps: in step S510 , obtaining second image data and a second numerical value corresponding to the second image data.

[0140] The second image data may include robot general data and task data.

[0141] In order to enable the language model to have basic general robot capabilities, open source robot data can be collected, mainly including robotic arm robot data, such as kuka, bridge, taco_play, and jaco_play data.

[0142] The task data can be robot simulation data that is freely generated using various robot simulation platforms and is consistent with specific tasks. The robot simulation platform can be UE5 (Unreal Engine 5) or other platforms, and this exemplary embodiment does not specifically limit this.

[0143] In step S520 , the second image data is input into the language model to be trained, so that the language model to be trained outputs a second value, so as to fine-tune the language model to be trained to obtain a trained language model.

[0144] In an optional embodiment, the second image data includes third image data and fourth image data, and the second numerical value includes a third numerical value and a fourth numerical value. Figure 6 shows a flow chart of a method for further training a language model. As shown in Figure 6, the method may include at least the following steps: in step S610, the third image data is input into the language model to be trained, so that the language model to be trained outputs the third numerical value, so as to perform initial fine-tuning on the language model to be trained to obtain an initially fine-tuned language model.

[0145] In an optional embodiment, the third value and the fourth value adopt a canonical representation of the reinforcement learning dataset.

[0146] Researchers propose the Reinforcement Learning Dataset (RLDS) and release a set of tools for recording, replaying, manipulating, annotating, and sharing data for sequential decision making, including offline reinforcement learning, apprenticeship learning, or imitation learning.

[0147] RLDS makes it easy to share datasets without losing any information (e.g., preserving the sequence of interactions rather than randomizing them) and is independent of the underlying raw format, allowing users to quickly test new algorithms on a wider range of tasks.

[0148] In addition, RLDS provides tools for collecting data generated by synthetic agents or humans, as well as tools for inspecting and processing the collected data.

[0149] Finally, integration with TensorFlow Dataset helps facilitate sharing of reinforcement learning datasets with the research community.

[0150] Algorithms used in reinforcement learning, offline reinforcement learning, or imitation learning may all use data in completely different formats. When the format of a dataset is unclear, errors can easily arise from misunderstanding the underlying data. RLDS makes the data format explicit by defining the content and meaning of each field in the dataset and provides tools for realigning and transforming it to fit the format required by any algorithm implementation.

[0151] To define the data format, RLDS leverages the standard structure inherent in reinforcement learning datasets, namely a sequence (episode) of interactions (steps) between an agent and an environment, where the agent can be a rule-based / automated controller, a formal planner, a human, an animal, or a combination of these.

[0152] Each of these steps contains the current observation, the action applied to the current observation, the reward obtained as a result of applying the action, and the discount obtained along with the reward. Steps also include additional information to indicate whether the step is the first or last step of the episode, or whether the observation corresponds to a terminal state. Each step and episode can also contain custom metadata that can be used to store environment-related or model-related data.

[0153] The third image data may be collected open source robot data, and the third value corresponding to the third image data may be a motion parameter of the robot data in the image, such as a position parameter, a rotation parameter, etc.

[0154] Since the third image data is a collected open source robot data set, the meaning or value range of each field of the third numerical value corresponding to the third image data is not uniform, so the third numerical value can be subjected to data cleaning processing.

[0155] In an optional embodiment, data cleaning is performed on the third value to obtain a cleaned third value.

[0156] In an optional embodiment, Figure 7 shows a flow chart of a method for data cleaning processing. As shown in Figure 7, the method may include at least the following steps: in step S710, the value range of the third value is unified according to the specification of the reinforcement learning data set to obtain a unified third value.

[0157] By analyzing the meaning of each field in the RLDS structure of the third value, it can be found that different open source robot datasets have different regulations on the value range of the same field. Therefore, the one with the most identical value ranges can be used as the value range to be unified, and the maximum and minimum values ​​of other value ranges that do not belong to this value range can be converted and unified to obtain the unified third value.

[0158] In addition, other unified methods may be selected for the value range, and this exemplary embodiment does not impose any special limitation on this.

[0159] In step S720, the representation of the third value is unified according to the specification of the reinforcement learning data set to obtain a unified third value.

[0160] By analyzing the meaning of each field in the RLDS structure of the third value, it can be found that different open source robot datasets have different representations of the same field. For example, some use -1 to represent the closed state and 0 to represent the open state, while others use 0 to represent the closed state and 1 to represent the open state, etc. Therefore, the one with the most expressions can be selected as the expression to be unified, and other expressions that do not use this expression can be adjusted to obtain the unified third value.

[0161] In addition, other unified methods may also be selected for the representation, and this exemplary embodiment does not impose any special limitation on this.

[0162] In step S730, the third value is eliminated according to the specification of the reinforcement learning data set to obtain the eliminated third value.

[0163] By analyzing the meaning of each field in the RLDS structure of the third value, it can be found that the open source robot dataset may contain no operations represented by 0. Therefore, the values ​​representing no operations in the third value can be eliminated to obtain the eliminated third value.

[0164] In addition, the third values ​​representing other operations may be eliminated, or the third values ​​representing no operations may be processed in other ways, which is not particularly limited in this exemplary embodiment.

[0165] In order to implement the training of the language model to be trained, the framework environment of transformers and DeepSpeed ​​can be deployed to provide support.

[0166] DeepSpeed ​​is an open-source deep learning optimization library developed by Microsoft that aims to improve the efficiency and scalability of large-scale model training. It accelerates training through a variety of techniques, including model parallelization, gradient accumulation, dynamic precision scaling, and local mode mixed precision.

[0167] DeepSpeed ​​also provides some auxiliary tools, such as distributed training management, memory optimization, and model compression, to help developers better manage and optimize large-scale deep learning training tasks.

[0168] Furthermore, DeepSpeed ​​is built on PyTorch, an open-source Python machine learning library, and can be migrated with minimal modifications. DeepSpeed ​​has been used in many large-scale deep learning projects, including language modeling, image classification, and object detection.

[0169] DeepSpeed ​​is a large-scale model training acceleration library located between the model training framework and the model, used to improve training, reasoning, etc.

[0170] DeepSpeed ​​mainly consists of the following three parts:

[0171] APIs (Application Programming Interfaces): These provide easy-to-use APIs. Training and inference models require only a few simple calls. The most important of these is the initialize API, which provides initialization methods and is used to initialize the engine and configure training parameters and optimization techniques. Configuration parameters are typically stored in a config.json file (a configuration file used to store application settings).

[0172] Runtime: This component is the core component for DeepSpeed ​​management, execution, and performance optimization. Examples include deploying training tasks to distributed devices, data partitioning, model partitioning, system optimization, fine-tuning, and fault detection. This component is implemented in Python.

[0173] OPS (Orbeon Presentation Server, a web framework): uses C++ and CUDA (Compute Unified Device Architecture, a computing platform) to implement the underlying kernel and optimize computing and communication.

[0174] The benefits of this architectural design include: it can perform two-part optimization (separate training and inference) on the training framework; compared with a tightly coupled structure, this structure can better utilize the entire ecosystem and is easier to maintain than a deep integration; it is independent of the infrastructure and users can choose their favorite platform.

[0175] Furthermore, the third image data is input into the language model to be trained to perform initial fine-tuning on the language model to be trained, so that the language model to be trained outputs a third value, thereby obtaining the language model after initial fine-tuning.

[0176] Specifically, supervised fine-tuning technology can be used to perform initial fine-tuning on the language model to be trained.

[0177] SFT (Supervised Fine-Tuning) refers to pre-training a neural network model on a source dataset, namely the source model. A new neural network model is then created, namely the target model. The target model copies all model designs and parameters of the source model except the output layer. These model parameters contain the knowledge learned on the source dataset, and this knowledge is also applicable to the target dataset. The output layer of the source model is closely related to the labels of the source dataset and is therefore not used in the target model. During fine-tuning, an output layer with an output size equal to the number of categories in the target dataset is added to the target model, and the model parameters of this layer are randomly initialized. When training the target model on the target dataset, it is trained from scratch up to the output layer, and the parameters of the remaining layers are fine-tuned based on the parameters of the source model.

[0178] Specifically, supervised fine-tuning includes the following steps:

[0179] Pre-training: Training a deep learning model on a large dataset, such as using self-supervised learning or unsupervised learning algorithms for pre-training.

[0180] Fine-tuning: Fine-tuning the pre-trained model using the training set for the target task. Typically, only a few layers of the pre-trained model are fine-tuned, such as the last few layers or some intermediate layers. During fine-tuning, the model is optimized using the backpropagation algorithm to achieve better performance on the target task.

[0181] Evaluation: Use the test set of the target task to evaluate the fine-tuned model to obtain the performance indicators of the model on the target task.

[0182] In addition, reward modeling techniques can also be used to perform initial fine-tuning on the language model to be trained.

[0183] The reward model takes a text sequence as input and generates a reward value that aligns with human preferences. The training data for building the reward model is typically the same data generated using different language models, followed by human scoring.

[0184] Therefore, the goal of a reward model is to simulate how humans score text. There are many strategies for building reward models, such as the most straightforward predictive annotation, which can output a score or Boolean value based on whether it's good or bad. Alternatively, there's the ranking of predictions, where the model predicts which of the two human-annotated results for each input text has a higher score.

[0185] Furthermore, human feedback reinforcement learning techniques can be used to perform initial fine-tuning on the language model being trained.

[0186] RLHF (Reinforcement learning with human feedback) uses reinforcement learning to optimize language models based on human feedback.

[0187] Human-feedback reinforcement learning is an advanced method for training AI systems that combines reinforcement learning with human feedback. It's a method for creating a more robust learning process by incorporating the wisdom and experience of human trainers into the model training process. The technique involves using human feedback to create a reward signal, which is then used through reinforcement learning to improve the model's behavior.

[0188] Reinforcement learning, simply put, is a process in which an AI agent learns to make decisions through interactions with its environment and feedback received in the form of rewards or penalties. The agent's goal is to maximize cumulative rewards over time. RLHF enhances this process by replacing or supplementing predefined reward functions with human-generated feedback, allowing the model to better capture complex human preferences and understanding.

[0189] The RLHF process can be divided into the following steps:

[0190] Initial model training: Initially, AI models are trained using supervised learning, where human trainers provide labeled examples of correct behaviors. The model learns to predict the correct action or output based on the given input.

[0191] Collecting human feedback: After the initial model is trained, human trainers provide feedback on the model's performance. They rank different model-generated outputs or behaviors based on quality or correctness. This feedback is used to create a reward signal for reinforcement learning.

[0192] Reinforcement Learning: The model is then fine-tuned using the Proximal Policy Optimization (PPO) algorithm or similar algorithms that incorporate human-generated reward signals. The model continuously improves its performance by learning from the feedback provided by the human trainer.

[0193] Iterative process: The process of collecting human feedback and improving the model through reinforcement learning is repeated, which will continuously improve the performance of the model.

[0194] In step S620 , the fourth image data is input into the language model after the initial fine-tuning, so that the language model after the initial fine-tuning outputs a fourth value, and the language model after the initial fine-tuning is subjected to advanced fine-tuning processing to obtain a trained language model.

[0195] The fourth image data may be robot simulation data that is freely generated using various robot simulation platforms and conforms to specific tasks, and the fourth numerical value corresponding to the fourth image data may be motion parameters of the robot data in the simulation data of the image, such as position parameters, rotation parameters, etc.

[0196] Since the fourth image data and the fourth numerical value are generated through a simulation platform, the specification of the fourth numerical value can be agreed upon before generation, without the need for data cleaning processing.

[0197] Furthermore, the fourth image data is input into the language model after the initial fine-tuning to perform advanced fine-tuning processing on the language model after the initial fine-tuning, so that the language model after the initial fine-tuning outputs a fourth value, thereby obtaining a trained language model.

[0198] Specifically, supervised fine-tuning technology, reward modeling technology, or human feedback reinforcement learning technology can be used to perform advanced fine-tuning on the language model after initial fine-tuning.

[0199] After obtaining the trained language model, the visual embedding vector and the text embedding vector may be input into the trained language model so that the trained language model outputs a target value.

[0200] In an optional embodiment, the target values ​​include: a first group of values, a second group of values ​​and a third group of values, and the motion parameters include: position parameters, rotation parameters and state parameters, the first group of values ​​represents the position parameters, the second group of values ​​represents the rotation parameters, and the third group of values ​​represents the state parameters.

[0201] The robot's movement parameters mainly include three parts: EE Position (position parameter), EE Rotation (rotation parameter) and Gripper Status (status parameter).

[0202] In the terminology of robotic arm robots, "EE" usually refers to "End-Effector".

[0203] The end effector is the component mounted at the farthest end of a manipulator or robot arm and is used to interact with the environment, such as grasping, moving objects, and manipulating tools. Therefore, "EE Position" and "EE Rotation" represent the position and rotation of the end effector, respectively.

[0204] Specifically, EE Position refers to the position coordinates of the end effector in space, typically expressed as values ​​in the x, y, and z axes. These coordinates can be defined based on different reference frames, such as a global reference frame (relative to the entire workspace) or a local reference frame (relative to a specific part, such as the base of the robot arm). Determining EE Position is crucial for precisely controlling the robot arm to place an object at the desired location.

[0205] EE Rotation describes the angle or orientation of the end effector's rotation around various axes. Rotation is typically represented using Euler angles (Yaw, Pitch, Roll), quaternions, or rotation matrices. Understanding EE Rotation is crucial for ensuring proper tool contact with the target object, especially when high-precision positioning is required.

[0206] Gripper Status describes the open or closed state of the end effector.

[0207] The first group of values ​​is used to represent position parameters, so it can be represented by three numbers, such as [32112, 32152, 32167]; the second group of values ​​is used to represent rotation parameters, so it can also be represented by three numbers, such as [32186, 32205, 32209]; the third group of values ​​is used to represent state parameters, so it can be represented by one number, such as 32258.

[0208] In step S140, the target value is decoded to obtain motion parameters, which are used to control the motion of the robot.

[0209] In an exemplary embodiment of the present disclosure, after the trained language model outputs a target value, the target value may be decoded to obtain motion parameters.

[0210] In an optional embodiment, Figure 8 shows a flow chart of a method for decoding a target value. As shown in Figure 8, the method may include at least the following steps: in step S810, obtain a first preset value, a second preset value, a first threshold value and a second threshold value corresponding to a first group of values, and calculate the first group of values ​​and the first preset value to obtain a first calculation result.

[0211] Among them, the first preset value can be the number of words that can be represented by the vocabulary of the language model. For example, when using the vocabulary of LLaMA (a large language model based on the transformer architecture), the original vocabulary of LLaMA is 32,000, so the first preset value can be 32,000. Then, the target value is equivalent to a new vocabulary added on the basis of the original vocabulary.

[0212] The second preset value can be considered to be discretized target values ​​output by the language model. Therefore, the target values ​​can be divided into N buckets, where N is the second preset value and can be set to 256.

[0213] The first threshold value may represent the maximum value of the value range of the first set of values, and is represented by HIGH. Generally, HIGH may be 0.2.

[0214] The second threshold value may represent the minimum value of the value range of the first set of values, and is represented by LOW. Generally, LOW may be -0.2.

[0215] Furthermore, before the calculation, the first set of values ​​is represented by F, and the first set of values ​​are all integers. The calculation of the first set of values ​​and the first preset value can be F=F-32000, which is equivalent to F being updated according to the calculation situation. In this case, F represents the first calculation result.

[0216] This calculation is equivalent to taking into account that the target value output by the language model is discretized. Therefore, the target value can be divided into N buckets, each bucket has an ID corresponding to the token ID of the language model. Therefore, the first calculation result F represents the bucket number.

[0217] In step S820, the first calculation result and the second preset value are calculated to obtain a second calculation result, and the second calculation result, the first threshold value and the second threshold value are calculated to obtain a position parameter.

[0218] After obtaining the first calculation result, the first calculation result and the second preset value can be calculated to obtain a second calculation result. Specifically, the first calculation result and the second preset value can be calculated in a manner of F=F / 256. In this case, F represents the second calculation result, which is a relative value used to indicate which bucket the target value is in.

[0219] Furthermore, the calculation of the second calculation result, the first threshold and the second threshold can be performed by first calculating F=F*(HIGH-LOW) and then calculating F=F+LOW to obtain the position parameter to obtain the actual distance the robot needs to move, which can be in the form of a floating-point value.

[0220] In an optional embodiment, Figure 9 shows a flow chart of another method for decoding a target value. As shown in Figure 9, the method may include at least the following steps: in step S910, obtain a first preset value, a second preset value, a third threshold value and a fourth threshold value corresponding to the second group of values, and calculate the second group of values ​​and the first preset value to obtain a third calculation result.

[0221] Among them, the first preset value can be the number of words that can be represented by the vocabulary of the language model. For example, when using the LLaMA vocabulary, the original LLaMA vocabulary is 32,000, so the first preset value can be 32,000. Then, the target value is equivalent to a new vocabulary added based on the original vocabulary.

[0222] The second preset value can be considered to be discretized target values ​​output by the language model. Therefore, the target values ​​can be divided into N buckets, where N is the second preset value and can be set to 256.

[0223] The third threshold value may represent the maximum value of the value range of the second set of values, and may also be represented by HIGH. Generally, HIGH may be 0.35.

[0224] The fourth threshold may represent the minimum value of the value range of the second set of values, and may also be represented by LOW. Generally, LOW may be -0.35.

[0225] Furthermore, before the calculation, the second set of values ​​is represented by F, and the second set of values ​​are all integers. The calculation of the second set of values ​​and the first preset value can be F=F-32000, which is equivalent to F being updated according to the calculation situation. In this case, F represents the third calculation result.

[0226] This calculation is equivalent to taking into account that the target value output by the language model is discretized. Therefore, the target value can be divided into N buckets, each bucket has an ID corresponding to the token ID of the language model. Therefore, the third calculation result F represents the bucket number.

[0227] In step S920 , the third calculation result and the second preset value are calculated to obtain a fourth calculation result, and the fourth calculation result, the third threshold value, and the fourth threshold value are calculated to obtain a rotation parameter.

[0228] After obtaining the third calculation result, the third calculation result and the second preset value can be calculated to obtain a fourth calculation result. Specifically, the third calculation result and the second preset value can be calculated in a manner of F=F / 256. In this case, F represents the fourth calculation result, which is a relative value used to indicate which bucket the target value is in.

[0229] Furthermore, the calculation of the fourth calculation result, the third threshold and the fourth threshold can be performed by first calculating F=F*(HIGH-LOW) and then calculating F=F+LOW to obtain the rotation parameter to obtain the actual angle of rotation of the robot, which can be in the form of a floating-point value.

[0230] In an optional embodiment, Figure 10 shows a flow chart of another method for decoding a target value. As shown in Figure 10, the method may include at least the following steps: in step S1010, obtain a third preset value corresponding to the third group of values, and calculate the third group of values ​​and the third preset value to obtain a fifth calculation result.

[0231] The third preset value may be a corresponding value in addition to the number of the first preset value and the second preset value, the number of the position parameter, and the number of the rotation parameter. For example, if the number of the first preset value is 32000, the number of the second preset value is 256, and the number of the position parameter and the rotation parameter is 6, then the third preset value may be 32263.

[0232] Furthermore, the third group of values ​​may be 32263 and 32264. When the third preset value is 32263 and the third group of values ​​is 32263, the calculation of the third group of values ​​and the third preset value may be the third group of values ​​32263 minus the third preset value 32263 to obtain a fifth calculation result of 0; and when the third preset value is 32263 and the third group of values ​​is 32264, the calculation of the third group of values ​​and the third preset value may be the third group of values ​​32264 minus the third preset value 32263 to obtain a fifth calculation result of 1.

[0233] In step S1020, the state parameter is determined according to the fifth calculation result.

[0234] When the fifth calculation result is 0, it means that the state parameter is 0, indicating that the state of the end effector of the robot is closed; when the fifth calculation result is 1, it means that the state parameter is 1, indicating that the state of the end effector of the robot is open.

[0235] Therefore, after decoding the target value to obtain motion parameters such as position parameters, rotation parameters, and state parameters, the robot can move according to the motion parameters.

[0236] In addition to the three sets of values ​​representing position, rotation, and state parameters, you can also set an additional set of values ​​to indicate whether the robot's current task is complete. For example, these values ​​could be 32265 and 32266. When 32265 transitions to 0, the task is in progress; when 32266 transitions to 1, the task is complete.

[0237] The following describes in detail the method for controlling a robot in an embodiment of the present disclosure in conjunction with an application scenario.

[0238] FIG11 shows a flow chart of a method for controlling a robot in an application scenario. As shown in FIG11 , a target image captured by a shooting device installed on the robot is obtained, and a target instruction for controlling the robot is obtained.

[0239] The shooting device may be a camera or other devices, and this exemplary embodiment does not specifically limit this.

[0240] When the robot is a robotic arm or the robot has a robotic arm device, the shooting device can be installed at any position of the robot, such as a wrist and a joint, etc., and this exemplary embodiment does not impose any special limitation on this.

[0241] The target image can include images taken at the current moment by cameras at different positions of the robot as the current image, and can also include N historical images taken before the current moment, where N can be equal to 2 or 3, etc., or other values ​​can be set according to actual conditions. This exemplary embodiment does not make any special limitations on this.

[0242] The target instruction may be an instruction issued by a user to control the action of the robot, such as "pick up the apple on the table".

[0243] In step S1110 , the ViT picture coding model.

[0244] After acquiring the target image, the target image can be input into a trained visual model, which can be a ViT picture coding model.

[0245] The ViT model may be openai-ViT or google-ViT, and the selectable scale may be 0.08B, 0.6B, 1B, etc., which is not particularly limited in this exemplary embodiment.

[0246] Therefore, after the target image is input into the trained ViT image coding model, the target image can be encoded using the trained visual model to output a first image vector.

[0247] ViT models can use open-source models to avoid starting from scratch and quickly improve training efficiency. Furthermore, the larger the scale of the ViT model, the better the results, especially in terms of model generalization. For objects or environments not seen in the training set, the model's performance will be significantly improved.

[0248] In step S1120, a linear layer.

[0249] After outputting the first image vector, a linear layer may be used to perform mapping processing on the first image vector to obtain a visual embedding vector.

[0250] In step S1130, Vision Embedding.

[0251] Generally, the visual embedding vector can be represented by Vision Embedding.

[0252] It is worth noting that the size of the visual embedding vector and the subsequent text embedding vector should match to adapt to the requirements of the language model.

[0253] In step S1140, Tokenizer.

[0254] The tokenization technology can be used to implement text vectorization processing of the target instruction to obtain a second text vector.

[0255] In step S1150, Text Embedding.

[0256] Furthermore, the second text vector is mapped through the Text Embedding layer to obtain a text embedding vector.

[0257] In step S1160, the LLM large language model is generated.

[0258] FIG12 shows a flow chart of a method for training a language model in an application scenario. As shown in FIG12 , in step S1210 , data is collected (open source general data & task simulation data).

[0259] In order to enable the language model to have basic general robot capabilities, open source robot data can be collected, mainly including robotic arm robot data.

[0260] FIG13 shows a schematic diagram of a data set of a manipulator robot in an application scenario. As shown in FIG13 , the data set of the manipulator robot may include kuka, bridge, taco_play, jaco_play, etc., and this exemplary embodiment does not specifically limit this.

[0261] In addition, task data can also be collected.

[0262] FIG14 illustrates a schematic diagram of an interface for collecting task data in an application scenario. As shown in FIG14 , the task data can be robot simulation data generated freely using various robot simulation platforms to meet specific tasks. The robot simulation platform can be UE5 or other platforms, and this exemplary embodiment does not specifically limit this.

[0263] The researchers proposed a reinforcement learning dataset and released a set of tools for recording, replaying, manipulating, annotating, and sharing data, so that open source robot data and task data can adopt the canonical representation of the reinforcement learning dataset.

[0264] FIG15 shows a schematic diagram of an interface represented by Google's RLDS specification in an application scenario. As shown in FIG15 , “open_gripper” represents a state parameter, “rotation_delta” represents a rotation parameter, and “world_vector” represents a position parameter.

[0265] In step S1220, data cleaning is performed (data format is unified and noise data is identified).

[0266] Since the meanings or value ranges of the various fields of the third value corresponding to the open-source robot data are not uniform, the third value may be subjected to data cleaning processing.

[0267] By analyzing the meaning of each field in the RLDS structure of the third value, it can be found that different open source robot datasets have different regulations on the value range of the same field. Therefore, the one with the most identical value ranges can be used as the value range to be unified, and the maximum and minimum values ​​of other value ranges that do not belong to this value range can be converted and unified to obtain the unified third value.

[0268] By analyzing the meaning of each field in the RLDS structure of the third value, it can be found that different open source robot datasets have different representations of the same field. For example, some use -1 to represent the closed state and 0 to represent the open state, while others use 0 to represent the closed state and 1 to represent the open state, etc. Therefore, the one with the most expressions can be selected as the expression to be unified, and other expressions that do not use this expression can be adjusted to obtain the unified third value.

[0269] By analyzing the meaning of each field in the RLDS structure of the third value, it can be found that the open source robot dataset may contain no operations represented by 0. Therefore, the values ​​representing no operations in the third value can be eliminated to obtain the eliminated third value.

[0270] In step S1230 , the general multimodal data model is pre-trained.

[0271] In order to implement the training of the language model to be trained, the framework environment of transformers and DeepSpeed ​​can be deployed to provide support.

[0272] Training the model based on general multimodal data can enable the model to have multimodal conversation capabilities.

[0273] Multimodal data can be obtained by collecting open-source image and text datasets. For example, the collected datasets can be VQA datasets.

[0274] Specifically, the open source image dataset may be the first image data, and the open source text dataset may be the first text data.

[0275] The first numerical value corresponding to the first data may be a numerical value used to represent a word in the first text data in a vocabulary of a language model.

[0276] The first text vector may be obtained after performing text vectorization and mapping processing on the first text data.

[0277] After obtaining the first image data and the first text vector, the first image data can be input into the visual model to be trained, and the first text vector can be input into the language model to be trained, so that the language model outputs a first numerical value, thereby completing the training of the visual model to be trained and obtaining a trained visual model.

[0278] In step S1240 , the robot general model is fine-tuned.

[0279] The open source robot data is input into the language model to be trained to perform initial fine-tuning on the language model to be trained, so that the language model to be trained outputs a third value, thereby obtaining the language model after initial fine-tuning.

[0280] The model is trained on open-source general robotics data, giving it a certain level of embodied intelligence.

[0281] In step S1250 , the task data model is fine-tuned.

[0282] The robot simulation data that meets specific tasks freely generated by various robot simulation platforms are input into the language model after initial fine-tuning, so as to perform advanced fine-tuning processing on the language model after initial fine-tuning, so that the language model after initial fine-tuning outputs a fourth value, thereby obtaining a trained language model.

[0283] The LLM model may be an LLaMA2 model, a Baichuan model, or a Qwen model, and the optional scale may be 7B, 13B, 34B, 70B, etc., which is not particularly limited in this exemplary embodiment.

[0284] Fine-tune the model on simulated data of a specific task so that the model can be competent for the specific task.

[0285] LLM models can use open-source models to avoid starting from scratch and quickly improve training efficiency. Furthermore, the larger the LLM model, the better the results, especially in terms of model generalization. For objects or environments not seen in the training set, the model's performance will be significantly improved.

[0286] This approach uses an end-to-end model combining large visual and language models for training and further enabling robot control. During training, training progresses from general multimodal data to general robot data and then to robot task data, ultimately enabling the robot to handle specific user tasks and improving the success rate of these tasks.

[0287] After obtaining the trained language model, the visual embedding vector and the text embedding vector may be input into the trained language model so that the trained language model outputs a target value.

[0288] Specifically, the target values ​​include: a first group of values, a second group of values ​​and a third group of values, and motion parameters include: position parameters, rotation parameters and state parameters. The first group of values ​​represents the position parameters, the second group of values ​​represents the rotation parameters, and the third group of values ​​represents the state parameters.

[0289] The robot's movement parameters mainly include three parts: EE Position, EE Rotation and Gripper Status.

[0290] In step S1170, Action.

[0291] After the trained language model outputs the target value, the target value can be decoded to obtain the motion parameter.

[0292] After decoding the target value to obtain motion parameters such as position parameters, rotation parameters, and state parameters, the robot can move according to the motion parameters.

[0293] In addition to the three sets of values ​​representing position, rotation, and state parameters, you can also set an additional set of values ​​to indicate whether the robot's current task is complete. For example, these values ​​could be 32265 and 32266. When 32265 transitions to 0, the task is in progress; when 32266 transitions to 1, the task is complete.

[0294] In the exemplary embodiments of the present disclosure, target images and target instructions are input into the visual model and language model, respectively, for automatic task recognition. Based on the output target values, the robot is driven to perform various actions to complete the user's instructions, making human-computer interaction more intelligent and free, meeting the needs and uses of various application scenarios, improving the success rate of specific task execution, expanding the scope of robot use, and optimizing the user experience. Furthermore, through the application of an end-to-end multimodal large model, the embodied intelligence model no longer relies on the pipeline model, greatly improving the accuracy, flexibility, and practicality of the embodied intelligence system, and further reducing the development cost, maintenance cost, and usage cost of the embodied intelligence system. This is of great significance for promoting the development and application of embodied intelligence systems.

[0295] In addition, in an exemplary embodiment of the present disclosure, a device for controlling a robot is also provided. FIG16 shows a schematic structural diagram of the device for controlling a robot. As shown in FIG16 , the device 1600 for controlling a robot may include: a data acquisition module 1610, a visual processing module 1620, a language processing module 1630, and a decoding processing module 1640.

[0296] The data acquisition module 1610 is configured to acquire a target image captured by a camera mounted on the robot and acquire a target instruction for controlling the robot;

[0297] A visual processing module 1620 is configured to input the target image into a trained visual model so that the trained visual model outputs a visual embedding vector;

[0298] The language processing module 1630 is configured to obtain a text embedding vector corresponding to the target instruction, and input the visual embedding vector and the text embedding vector into a trained language model so that the trained language model outputs a target value;

[0299] The decoding processing module 1640 is configured to decode the target value to obtain motion parameters, and the motion parameters are used to control the motion of the robot.

[0300] In an exemplary embodiment of the present invention, the target picture includes: a current picture captured by the shooting device at a current moment, and historical pictures captured by the shooting device before the current moment.

[0301] In an exemplary embodiment of the present invention, the device 1600 for controlling a robot further includes:

[0302] a first acquisition module configured to acquire first data and a first value corresponding to the first data, and acquire a first text vector corresponding to the first text data, wherein the first data includes first image data and first text data;

[0303] The first training module is configured to input the first image data into the visual model to be trained, and input the first text vector into the language model to be trained to obtain a first value, so that the training of the visual model to be trained is completed to obtain a trained visual model.

[0304] In an exemplary embodiment of the present invention, the visual processing module 1620 includes:

[0305] An image output submodule is configured to input the target image into a trained visual model so that the trained visual model outputs a first image vector;

[0306] The first mapping submodule is configured to perform mapping processing on the first image vector through a linear layer to obtain a visual embedding vector.

[0307] In an exemplary embodiment of the present invention, the language processing module 1630 includes:

[0308] an instruction processing submodule, configured to perform text vectorization on the target instruction to obtain a second text vector;

[0309] The second mapping submodule is configured to perform mapping processing on the second text vector through a text embedding layer to obtain a text embedding vector.

[0310] In an exemplary embodiment of the present invention, the device 1600 for controlling a robot includes:

[0311] a second acquisition module, configured to acquire second image data and a second value corresponding to the second image data;

[0312] The second training module is configured to input the second image data into the language model to be trained, so that the language model to be trained outputs the second value, so as to fine-tune the language model to be trained to obtain a trained language model.

[0313] In an exemplary embodiment of the present invention, the second picture data includes third picture data and fourth picture data, and the second value includes a third value and a fourth value;

[0314] The second training module includes:

[0315] a first fine-tuning submodule configured to input the third image data into a language model to be trained, causing the language model to be trained to output the third value, thereby performing an initial fine-tuning process on the language model to be trained to obtain an initially fine-tuned language model;

[0316] The second fine-tuning submodule is configured to input the fourth image data into the language model after initial fine-tuning, so that the language model after initial fine-tuning outputs a fourth value, so as to perform advanced fine-tuning on the language model after initial fine-tuning to obtain a trained language model.

[0317] In an exemplary embodiment of the present invention, the device 1600 for controlling a robot further includes:

[0318] The data cleaning module is configured to perform data cleaning on the third value to obtain a cleaned third value.

[0319] In an exemplary embodiment of the present invention, the third value and the fourth value adopt a canonical representation of a reinforcement learning dataset.

[0320] In an exemplary embodiment of the present invention, the data cleaning module includes:

[0321] A first cleaning submodule is configured to unify the value range of the third value according to the specification of the reinforcement learning data set to obtain a unified third value; and / or

[0322] A second cleaning submodule is configured to unify the representation of the third value according to the specification of the reinforcement learning data set to obtain a unified third value; and / or

[0323] The third cleaning submodule is configured to perform a elimination process on the third value according to the specification of the reinforcement learning data set to obtain a eliminated third value.

[0324] In an exemplary embodiment of the present invention, the target values ​​include: a first group of values, a second group of values ​​and a third group of values, and the motion parameters include: position parameters, rotation parameters and state parameters, the first group of values ​​represents the position parameters, the second group of values ​​represents the rotation parameters, and the third group of values ​​represents the state parameters.

[0325] In an exemplary embodiment of the present invention, the decoding processing module 1640 includes:

[0326] a first calculation submodule configured to obtain a first preset value, a second preset value, a first threshold value, and a second threshold value corresponding to the first set of values, and calculate the first set of values ​​and the first preset value to obtain a first calculation result;

[0327] The position determination submodule is configured to calculate the first calculation result and the second preset value to obtain a second calculation result, and to calculate the second calculation result, the first threshold value, and the second threshold value to obtain a position parameter.

[0328] In an exemplary embodiment of the present invention, the decoding processing module 1640 includes:

[0329] a second calculation submodule configured to obtain a first preset value, a second preset value, a third threshold value, and a fourth threshold value corresponding to the second set of values, and calculate the second set of values ​​and the first preset value to obtain a third calculation result;

[0330] The rotation determination submodule is configured to calculate the third calculation result and the second preset value to obtain a fourth calculation result, and calculate the fourth calculation result, the third threshold and the fourth threshold to obtain a rotation parameter.

[0331] In an exemplary embodiment of the present invention, the decoding processing module 1640 includes:

[0332] a third calculation submodule configured to obtain a third preset value corresponding to the third set of numerical values, and calculate the third set of numerical values ​​and the third preset value to obtain a fifth calculation result;

[0333] The state determination submodule is configured to determine the state parameter according to the fifth calculation result.

[0334] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0335] FIG17 is a block diagram of an electronic device 1700 according to an exemplary embodiment. As shown in FIG17 , the electronic device 1700 may include a processor 1701 and a memory 1702. The electronic device 1700 may also include one or more of a multimedia component 1703, an input / output (I / O) interface 1704, and a communication component 1705.

[0336] The processor 1701 is used to control the overall operation of the electronic device 1700 to complete all or part of the steps in the above-mentioned method of controlling a robot. The memory 1702 is used to store various types of data to support the operation of the electronic device 1700. Such data may include, for example, instructions for any application or method operating on the electronic device 1700, as well as application-related data, such as contact information, sent and received messages, pictures, audio, video, etc. The memory 1702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 1703 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in the memory 1702 or transmitted via the communication component 1705. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 1704 provides an interface between the processor 1701 and other interface modules, which may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 1705 is used for wired or wireless communication between the electronic device 1700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more thereof, is not limited here. Therefore, the corresponding communication component 1705 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0337] In an exemplary embodiment, the electronic device 1700 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the above-mentioned method of controlling a robot.

[0338] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided. When executed by a processor, the program instructions implement the steps of the above-described method for controlling a robot. For example, the computer-readable storage medium may be the aforementioned memory 1702 including the program instructions. The program instructions may be executed by the processor 1701 of the electronic device 1700 to perform the above-described method for controlling a robot.

[0339] Figure 18 is a block diagram of an electronic device 1800 according to an exemplary embodiment. For example, electronic device 1800 can be provided as a server. Referring to Figure 18 , electronic device 1800 includes one or more processors 1822 and a memory 1832 for storing a computer program executable by processor 1822. The computer program stored in memory 1832 may include one or more modules, each corresponding to a set of instructions. Furthermore, processor 1822 may be configured to execute the computer program to perform the aforementioned method of controlling a robot.

[0340] In addition, the electronic device 1800 may further include a power supply component 1826 and a communication component 1850. The power supply component 1826 may be configured to perform power management of the electronic device 1800, and the communication component 1850 may be configured to implement communication, such as wired or wireless communication, of the electronic device 1800. In addition, the electronic device 1800 may further include an input / output (I / O) interface 1858. The electronic device 1800 may operate based on an operating system stored in the memory 1832.

[0341] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided. When executed by a processor, the program instructions implement the steps of the above-described method for controlling a robot. For example, the non-transitory computer-readable storage medium may be the aforementioned memory 1832 including the program instructions. The program instructions may be executed by the processor 1822 of the electronic device 1800 to perform the above-described method for controlling a robot.

[0342] In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program executable by a programmable device, and has code portions for executing the above-mentioned method of controlling a robot when executed by the programmable device.

[0343] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings. However, the present disclosure is not limited to the specific details of the above embodiments. Within the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all fall within the scope of protection of the present disclosure.

[0344] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0345] In addition, the various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A method for controlling a robot, characterized in that: The method comprises: Obtaining a target image captured by a camera mounted on the robot, and obtaining a target instruction for controlling the robot; Inputting the target image into a trained visual model so that the trained visual model outputs a visual embedding vector; Obtaining a text embedding vector corresponding to the target instruction, and inputting the visual embedding vector and the text embedding vector into a trained language model so that the trained language model outputs a target value; The target value is decoded to obtain motion parameters, and the motion parameters are used to control the motion of the robot.

2. The method for controlling a robot according to claim 1, wherein: The target picture includes: a current picture captured by the shooting device at the current moment, and historical pictures captured by the shooting device before the current moment.

3. The method for controlling a robot according to claim 1, wherein: Before inputting the target image into the trained visual model, the method further includes: Acquire first data and a first value corresponding to the first data, and acquire a first text vector corresponding to the first text data, where the first data includes first image data and first text data; The first image data is input into the visual model to be trained, and the first text vector is input into the language model to be trained to obtain a first value, so that the training of the visual model to be trained is completed to obtain a trained visual model.

4. The method for controlling a robot according to claim 1, wherein: Inputting the target image into a trained visual model so that the trained visual model outputs a visual embedding vector includes: Inputting the target image into a trained visual model so that the trained visual model outputs a first image vector; The first image vector is mapped through a linear layer to obtain a visual embedding vector.

5. The method for controlling a robot according to claim 1, wherein: The obtaining of a text embedding vector corresponding to the target instruction includes: Performing text vectorization on the target instruction to obtain a second text vector; The second text vector is mapped by a text embedding layer to obtain a text embedding vector.

6. The method for controlling a robot according to claim 1, wherein: Before inputting the visual embedding vector and the text embedding vector into the trained language model, the method further includes: Acquire second image data and a second value corresponding to the second image data; The second image data is input into the language model to be trained, so that the language model to be trained outputs the second value, so as to fine-tune the language model to be trained to obtain a trained language model.

7. The method for controlling a robot according to claim 6, wherein: The second picture data includes third picture data and fourth picture data, and the second value includes a third value and a fourth value. Inputting the second image data into the language model to be trained, causing the language model to be trained to output the second value, and fine-tuning the language model to be trained to obtain a trained language model, includes: Inputting the third image data into a language model to be trained, so that the language model to be trained outputs the third value, and performing initial fine-tuning on the language model to be trained to obtain an initially fine-tuned language model; The fourth image data is input into the language model after initial fine-tuning, so that the language model after initial fine-tuning outputs a fourth value, and the language model after initial fine-tuning is advanced fine-tuned to obtain a trained language model.

8. The method for controlling a robot according to claim 7, wherein: Before inputting the third image data into the language model to be trained, the method further includes: Performing data cleaning on the third value to obtain a cleaned third value.

9. The method for controlling a robot according to claim 8, wherein: The third value and the fourth value adopt a standard representation of the reinforcement learning dataset.

10. The method for controlling a robot according to claim 9, wherein: The performing data cleaning on the third value to obtain a cleaned third value includes: Unifying the value range of the third value according to the specification of the reinforcement learning dataset to obtain a unified third value; and / or Unifying the representation of the third value according to the specification of the reinforcement learning dataset to obtain a unified third value; and / or The third value is eliminated according to the specification of the reinforcement learning data set to obtain the eliminated third value.

11. The method for controlling a robot according to claim 1, wherein: The target values ​​include: a first group of values, a second group of values ​​and a third group of values; the motion parameters include: position parameters, rotation parameters and state parameters; the first group of values ​​represents the position parameters, the second group of values ​​represents the rotation parameters, and the third group of values ​​represents the state parameters.

12. The method for controlling a robot according to claim 11, wherein: The decoding of the target value to obtain the motion parameter includes: Obtaining a first preset value, a second preset value, a first threshold value, and a second threshold value corresponding to the first set of values, and calculating the first set of values ​​and the first preset value to obtain a first calculation result; The first calculation result and the second preset value are calculated to obtain a second calculation result, and the second calculation result, the first threshold value, and the second threshold value are calculated to obtain a position parameter.

13. The method for controlling a robot according to claim 11, wherein: The decoding of the target value to obtain the motion parameter includes: Obtaining a first preset value, a second preset value, a third threshold value, and a fourth threshold value corresponding to the second set of values, and calculating the second set of values ​​and the first preset value to obtain a third calculation result; The third calculation result and the second preset value are calculated to obtain a fourth calculation result, and the fourth calculation result, the third threshold value, and the fourth threshold value are calculated to obtain a rotation parameter.

14. The method for controlling a robot according to claim 11, wherein: The decoding of the target value to obtain the motion parameter includes: Obtaining a third preset value corresponding to the third set of values, and calculating the third set of values ​​and the third preset value to obtain a fifth calculation result; The state parameter is determined according to the fifth calculation result.

15. A device for controlling a robot, characterized in that: include: a data acquisition module configured to acquire a target image captured by a camera mounted on the robot and to acquire a target instruction for controlling the robot; a visual processing module configured to input the target image into a trained visual model so that the trained visual model outputs a visual embedding vector; a language processing module configured to obtain a text embedding vector corresponding to the target instruction, and input the visual embedding vector and the text embedding vector into a trained language model so that the trained language model outputs a target value; The decoding processing module is configured to decode the target value to obtain motion parameters, and the motion parameters are used to control the motion of the robot.

16. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

17. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 14.

Images