A method performed by an electronic device and an electronic device

Abstract

Embodiments of the present disclosure provide a method and device executed by an electronic device, relating to the field of artificial intelligence. The method comprises: receiving a first instruction for moving a target object; determining, based on a scene image and the first instruction, a visual feature, a first hidden feature and a second hidden feature through a feature extraction network in a first AI model; determining, based on the first hidden feature, an action type associated with the target object through a selection module in the first AI model; selecting a target action model from a candidate action model set in the first AI model based on the action type; and determining, based on the second hidden feature and the visual feature, an action sequence associated with the target object through the target action model. The implementation improves the accuracy of predicting robot actions and improves the performance of robot operations. Optionally, the above-mentioned method executed by the electronic device can be executed using an artificial intelligence model.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of robotics, and more particularly to methods performed by electronic devices and electronic devices. Background Technology

[0002] Enabling robots to perform different tasks in different environments is a core challenge in robotics. Traditional robot learning methods typically develop strategies by curating datasets tailored to a specific robot and its specific tasks. In recent years, VLA (Vision Language Action) models have attracted considerable attention due to their ability to extend pre-trained visual language models to the robotics field. Summary of the Invention

[0003] The embodiments of this disclosure present methods and apparatus.

[0004] In a first aspect, embodiments of this disclosure provide a method executed by an electronic device, comprising: receiving a first instruction for moving a target object; determining visual features, a first hidden feature, and a second hidden feature based on a scene image and the first instruction, through a feature extraction network in a first AI model; determining an action type associated with the target object based on the first hidden feature, through a selection module in the first AI model; selecting a target action model from a set of candidate action models in the first AI model based on the action type; and determining an action sequence associated with the target object based on the second hidden feature and the visual features, through the target action model.

[0005] In some embodiments, the candidate action model set includes a first model associated with arm movements and a second model associated with gripper operations.

[0006] In some embodiments, based on a first hidden feature, the action type associated with the target object is determined by a selection module in a first AI model, including: based on the first hidden feature, outputting a predicted value of the action type associated with the target object by a selection module in a first AI model; determining the action type as an arm movement in response to a predicted value being less than a first threshold; and determining the action type as a grasper operation in response to a predicted value being greater than or equal to the first threshold.

[0007] In some embodiments, the visual features include global visual features and local visual features; and based on the second hidden features and the visual features, determining the action sequence associated with the target object through a target action model includes: in response to the action type being arm movement, determining the action sequence associated with the target object through a first model based on the second hidden features and global visual features; or in response to the action type being a grasper operation, determining the action sequence associated with the target object through a second model based on the second hidden features and local visual features.

[0008] In some embodiments, the visual features include global visual features and local visual features; and based on the second hidden features and the visual features, determining the action sequence associated with the target object through a target action model includes: in response to the action type being arm movement, determining the action sequence associated with the target object through a first model based on random noise, the second hidden features and the global visual features; or in response to the action type being a grasper operation, determining the action sequence associated with the target object through a second model based on random noise, the second hidden features and the local visual features.

[0009] In some embodiments, the feature extraction network includes an encoder, a visual model, and a large language model; and based on a scene image and a first instruction, the feature extraction network in the first AI model determines visual features, a first hidden feature, and a second hidden feature, including: determining language features based on the first instruction using the encoder; determining a first visual feature and a second visual feature based on the scene image using the visual model, wherein the first visual feature is used to characterize the basic structure and local information of the scene image, and the second visual feature is used to characterize the semantic information of the scene image; and determining the first hidden feature and the second hidden feature based on the first visual feature and the language feature using the large language model.

[0010] In some embodiments, the large language model is an LLAMA-2 model, and the first hidden feature and the second hidden feature are respectively derived from the hidden features of the second Transformer layer and the hidden features of the last Transformer layer of the LLAMA-2 model.

[0011] In some embodiments, determining the action sequence associated with the target object based on the second hidden feature and the visual feature through the target action model includes: determining the action sequence associated with the target object based on the second hidden feature and the second visual feature through the target action model.

[0012] In some embodiments, the feature extraction network includes the DINOv2 model and the SigLIP model.

[0013] In a second aspect, embodiments of this disclosure provide a robot system, including: a server configured to execute the method of either the first aspect or the second aspect, and output an action sequence; and a robot configured to receive the action sequence and perform actions according to the action sequence.

[0014] Thirdly, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more computer programs stored thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors perform the method as described in any one of the first or second aspects.

[0015] Fourthly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of the first or second aspects.

[0016] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, implements the method as described in any one of the first or second aspects.

[0017] The methods and electronic devices executed by electronic devices provided in the embodiments of this disclosure, through a detailed analysis of existing robot operation processes, reveal numerous differences between robot arm movements and gripper operations in terms of the fineness of the movements, the image attention areas required for the operation, and the proportion of different actions in the dataset. Based on these differences, this disclosure designs a DAM-VLA (VLM-based Dynamic Action Model) framework based on VLM (Vision Language Models), dynamically combining the inherent reasoning capabilities of VLM with the precise operational capabilities of the diffusion action model. More specifically, this disclosure utilizes the inherent reasoning capabilities of VLM to dynamically distinguish different action models, and then uses the corresponding action model to complete specific and precise operations.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] Figure 1 This is an exemplary system architecture diagram to which one embodiment of this disclosure can be applied;

[0020] Figure 2 This is a diagram analyzing the differences between the robot arm's movements and the gripper's operations;

[0021] Figure 3 This is a flowchart of one embodiment of a method performed by an electronic device according to the present disclosure;

[0022] Figure 4 This is a flowchart of yet another embodiment of a method performed by an electronic device according to the present disclosure;

[0023] Figure 5 This is a schematic diagram of an application scenario of the method performed by an electronic device according to this disclosure;

[0024] Figure 6This is a schematic diagram of a dynamic motion integration solution;

[0025] Figure 7 This is a structural schematic diagram of the robot system according to this disclosure;

[0026] Figure 8 This is a schematic diagram of the structure of a computer system suitable for implementing embodiments of the present disclosure. Detailed Implementation

[0027] The following description, with reference to the accompanying drawings, is provided to aid in a thorough understanding of the various embodiments of this disclosure as defined by the claims and their equivalents. This description includes various specific details to aid understanding but should be considered exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of this disclosure. Furthermore, for clarity and brevity, descriptions of well-known functions and structures may be omitted.

[0028] The terms and wording used in the following description and claims are not limited to their dictionary meanings, but are merely used by the inventors to enable a clear and consistent understanding of this disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of this disclosure is for illustrative purposes only and not for limiting the purpose of this disclosure as defined in the appended claims and their equivalents.

[0029] It should be understood that the singular forms of “a,” “an,” and “the” can also include plural references unless the context clearly indicates otherwise. Thus, for example, the reference to “component surface” includes referring to one or more such surfaces. When we say that an element is “connected” or “coupled” to another element, the element can be directly connected or coupled to the other element, or it can mean that the element and the other element are connected through an intermediate element. Furthermore, the use of “connected” or “coupled” herein can include wireless connections or wireless couplings.

[0030] The terms “comprising” or “may include” refer to the presence of a corresponding disclosed function, operation, or component that may be used in the various embodiments of this disclosure, rather than limiting the presence of one or more additional functions, operations, or features. Furthermore, the terms “comprising” or “having” may be interpreted as indicating certain characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof, but should not be construed as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof.

[0031] The term "or" as used in the various embodiments of this disclosure includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B. When describing multiple (two or more) items, if the relationship between the multiple items is not explicitly defined, the multiple items may refer to one, more, or all of the multiple items. For example, the description "parameter A includes A1, A2, A3" can be implemented as parameter A includes A1 or A2 or A3, or it can be implemented as parameter A includes at least two of the three items A1, A2, and A3.

[0032] Unless otherwise defined, all terms used in this disclosure (including technical or scientific terms) have the same meaning as understood by one of those skilled in the art as described herein. Common terms as defined in dictionaries are to be interpreted as having a meaning consistent with the context in the relevant technical field and should not be interpreted ideally or overly formally unless expressly defined in this disclosure.

[0033] At least some of the functions of the device or electronic device provided in this disclosure embodiment can be implemented by an AI model, such as implementing at least one module of a plurality of modules of the device or electronic device by an AI model. AI-related functions can be executed by non-volatile memory, volatile memory, and a processor.

[0034] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as central processing unit (CPU), application processor (AP), etc., or pure graphics processing unit, such as graphics processing unit (GPU), vision processing unit (VPU), and / or AI-specific processors, such as neural processing unit (NPU).

[0035] The one or more processors control the processing of input data based on predefined operating rules or artificial intelligence (AI) models stored in non-volatile and volatile memory. These predefined operating rules or AI models are provided through training or learning.

[0036] Here, "providing through learning" refers to obtaining predefined operating rules or an AI model with desired characteristics by applying a learning algorithm to multiple learning datasets. This learning can be performed within the device or electronic device itself, in which the AI ​​is executed according to the embodiment, and / or can be implemented via a separate server / system.

[0037] AI models can contain multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network computations by calculating the input data of that layer (such as the computation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q-networks.

[0038] A learning algorithm is a method of training a predetermined target device (e.g., a robot) using multiple learning data sets to enable, allow, or control the target device to make determinations or predictions. Examples of such learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0039] The methods provided in this disclosure may relate to one or more fields in the technical fields of speech, language, image, video, or data intelligence.

[0040] Optionally, in the context of speech or language, according to this disclosure, in the training of a robot action prediction model and the method for predicting robot actions executed by an electronic device, language instructions and images are used as training samples to train the robot action prediction model. Speech signals, as analog signals, can be received via a speech acquisition device (e.g., a microphone), and the speech portions can be converted into computer-readable text using an Automatic Speech Recognition (ASR) model. The user's utterance intent can be obtained by interpreting the converted text using a Natural Language Understanding (NLU) model. The ASR model or NLU model can be an artificial intelligence model. The artificial intelligence model can be processed by a dedicated artificial intelligence processor designed in a hardware architecture specified for processing the artificial intelligence model. The artificial intelligence model can be obtained through training. Here, "obtained through training" means obtaining a predefined operating rule or artificial intelligence model configured to perform desired features (or purposes) by training a basic artificial intelligence model with multiple training data using a training algorithm. Language understanding is a technique for recognizing and applying / processing human language / text, including, for example, natural language processing, machine translation, dialogue systems, question answering, or speech recognition / synthesis.

[0041] Optionally, in the context of images or videos, according to this disclosure, in the method of training a robot motion prediction model and predicting robot motion executed in an electronic device, image data can be used as input data to the artificial intelligence model, and the predicted robot motion can be used as output data. The artificial intelligence model can be obtained through training. Here, "obtained through training" means obtaining a predefined operating rule or artificial intelligence model configured to perform desired features (or purposes) by training a basic artificial intelligence model with multiple training data using a training algorithm. The methods of this disclosure can relate to the field of visual understanding in artificial intelligence technology, which is a technology for recognizing and processing things like human vision, and includes, for example, object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / localization, or image enhancement.

[0042] Optionally, in the field of data intelligence processing, according to this disclosure, in the training robot motion prediction model and the method for predicting robot motion executed in an electronic device, the method for reasoning or predicting robot motion can use an artificial intelligence model to recommend / execute the training robot motion prediction model and the prediction of robot motion using [the data used to train the robot motion prediction model and predict robot motion]. The processor of the electronic device can perform preprocessing operations on the data to transform it into a form suitable for use as input to the artificial intelligence model. The artificial intelligence model can be obtained through training. Here, "obtained through training" means obtaining a predefined operating rule or artificial intelligence model configured to perform desired features (or purposes) by training a basic artificial intelligence model with multiple training data through a training algorithm. Reasoning and prediction are techniques for logical reasoning and prediction based on determined information, including, for example, knowledge-based reasoning, optimization prediction, preference-based planning, or recommendation.

[0043] The following description of several optional embodiments illustrates the technical solutions of this disclosure and the technical effects produced by these solutions. It should be noted that the following embodiments can be referenced, learned from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0044] Figure 1 An exemplary system architecture is shown that embodiments of the methods performed by electronic devices according to this disclosure can be applied.

[0045] like Figure 1As shown, the robot motion prediction model (also known as the first AI model) includes a feature extraction network, a motion model selection module (referred to as the selection module), and a dynamic motion model (i.e., the target motion model selected from the set of candidate motion models). The feature extraction network may include an encoder, a vision model, and a large language model. The dynamic motion model illustrated in this paper may include an arm motion model and a gripper operation model, named the first model and the second model, respectively. In practice, many more types of motion models can be used, and this is not limited here.

[0046] The robot motion prediction model disclosed herein enables different robots to physically perform different tasks simultaneously upon receiving a third-view RGB image observation o and a verbal instruction l. Formally, given the verbal instruction l at time t and the visual observation o... t Robot motion prediction model π predicts time motion sequences [a] t ,a t+1 ,…,a t+N ]=π(l,o t In this disclosure, the action space a t =[Δ{x},Δθ,grip] is the action of a gripper with 7 degrees of freedom (DoF). Here, Δx is the relative translational offset of the gripper, Δθ represents the rotational change, and grip∈0,1 represents the open / closed state of the gripper.

[0047] The architecture comprises five key components: 1) An encoder, used to encode user language commands (e.g., voice commands), which can be directly encoded or encoded after being converted into text. 2) A vision model, extracting visual features from scene images of the application robot. Visual features can include high-level and low-level visual features, where low-level features can include global and local visual features. For example, information observed in the current third-view RGB image is encoded as visual markers (high-level features), class markers (i.e., global visual features), and register markers (i.e., local visual features); 3) A large language model, which combines visual markers with encoded language commands to generate inference latent features and cognitive latent features; 4) An action model selection module, used to generate estimated weights for action types based on inference latent features and input them into the dynamic action model; 5) A dynamic action model, which dynamically executes different action models by combining visual class markers or register markers with cognitive latent features of the VLM to predict temporal action sequences.

[0048] like Figure 2 As shown, a detailed analysis of the existing robot operation process was conducted, revealing some differences between the arm movements and the gripper operations during robot operation:

[0049] 1) The required fineness of motion. In the task of placing a carrot on a plate, the robotic arm moves relatively freely and can reach the carrot via any path. In contrast, the carrot gripper's operation is not free; it requires a precise gripping posture.

[0050] 2) Image attention regions required for the action. Arm movements require more global attention, while grasper operations require more local attention.

[0051] 3) The proportion of different actions in the dataset. Compared with the grasper operation, the proportion of arm movement in the dataset is several times that of the grasper operation.

[0052] Based on these differences, this disclosure proposes a Dynamic Action Model-VLA (DAM-VLA) framework based on VLM, which dynamically combines the inherent reasoning capabilities of VLM with the precise operational capabilities of the diffusion action model. More specifically, this disclosure utilizes the inherent reasoning capabilities of VLM to dynamically distinguish between different action models—arm motion models and grasper manipulation models—and then uses the corresponding action models to perform specific and precise operations. The main contributions are as follows:

[0053] 1) This disclosure proposes an action model selection module. It utilizes the inherent reasoning ability of VLM to learn weights, thereby dynamically reasoning about different action types.

[0054] 2) This disclosure proposes a dynamic motion model. It combines high-level inference information from the VLM with low-level visual detail information into the motion model. More importantly, this disclosure designs different visual labels for different motion models: an arm motion model and a grasper manipulation model. For example, the arm motion model focuses more on global visual features, while the grasper manipulation model focuses more on local visual features.

[0055] 3) This disclosure designs a dynamic motion integration scheme by combining trajectory weights and action block weights, enabling the network to more robustly and accurately distinguish between the arm motion model and the grasper operation model.

[0056] 4) In the SIMPLER environment, the DAM-VLA method achieved the best average performance compared to the current best VLA method on the WindowsX robot and Google robot, while demonstrating stable operation capabilities in fine-grained manipulation, robot and object with various pose changes and environmental changes.

[0057] Continue to refer to Figure 3 The diagram illustrates a flow 300 of an embodiment of a method performed by an electronic device according to the present disclosure. The method performed by the electronic device includes the following steps:

[0058] Step 301: Receive a first instruction for moving the target object;

[0059] In this embodiment, the execution subject of the method executed by the electronic device may be... Figure 7 The server shown can also be a robot. The server can send the trained robot motion prediction model to the robot, which will then execute steps 302-305. The server can also execute steps 302-305 locally.

[0060] The robot can receive a first instruction from the user, sent via voice or remote control signal, to move the object. This first instruction can be a verbal command or other type of instruction. For example, the robot should perform a certain action to place a carrot on a plate. If steps 302-305 are executed by the server, the robot sends the verbal instruction to the server. The server can then notify a third-party camera to acquire a scene image of the object. The scene image can also be captured in real-time, without being triggered by the first instruction. Based on the verbal instruction, the server can recognize the user's intent and determine the target objects that need to be included in the scene image, such as the robot, the carrot, or the plate. The server can control the third-party camera to adjust its direction, angle, and distance from the target object to ensure that a scene image containing the target object is captured. If steps 302-305 are executed by the robot, the robot processes the verbal instruction locally and acquires a scene image of the object via its local camera or by notifying a third-party camera.

[0061] In this embodiment, the robot motion prediction model includes a feature extraction network, a motion model selection module, and a dynamic motion model. The robot motion prediction model can employ the methods described below. Figure 4 The training was performed using the method described in the illustrated embodiment. For a detailed training process, please refer to [link / reference needed]. Figure 4 The relevant descriptions of the embodiments will not be repeated here.

[0062] Step 302: Based on the scene image and the first instruction, determine the visual features, the first hidden feature and the second hidden feature through the feature extraction network in the first AI model;

[0063] In this embodiment, the feature extraction network can be a neural network used to extract features from the first instruction and the image. These features may include visual features, a first hidden feature, and a second hidden feature.

[0064] like Figure 5 As shown, the visual model processes the third-view RGB image input into a set of perceptual labels (i.e., visual features), where the visual features include not only the visual label f vis It also includes the class tag f cls and register flag f regThe visual model can consist of powerful visual transformers DINOv2 and SigLIP, pre-trained on internet-scale image data to capture rich low-level visual features and high-level semantic understanding of observations. At each time step t, the image observation o t The inputs are fed into DINOv2 and SigLIP, respectively generating two downsampled feature maps. and These feature maps are then concatenated along the channel dimension, passed through a linear projection layer, and serialized into a set of visual perceptual labels containing visual labels f. vis =[v1,v2,…,v NV ], class tag f cls and register flag f reg .

[0065] The large language model is responsible for integrating visual information and language instructions, and estimating inference tokens. Here, the LLAMA-2 model can be used as the backbone model. The language instruction l is converted into a set of language tokens f by the LLAMA-2 generator. lan =[l1,l2,…,l NL These tags f lan Then with visual marker f vis This is concatenated with an additional learnable inference tag and processed by a large language model using a causal attention mechanism. The final output is the cognitive hidden feature f. cog (i.e., the second hidden feature) and inference hidden feature f rea (i.e., the first hidden feature). f rea and f cog These are the hidden features from the second and last Transformer layers of the large language model, respectively. Cognitive hidden features f cog This serves as a condition for subsequent action model predictions. Inference hidden features f rea As a feature of the subsequent action selection module, it is used to dynamically infer different types of actions.

[0066] Step 303: Based on the first hidden feature, determine the action type associated with the target object through the selection module in the first AI model;

[0067] In this embodiment, the action model selection module can be a classifier. For example... Figure 5 As shown, the action model selection module utilizes the inference capabilities and truth weights (i.e., action type labels) of the VLM. Self-supervised learning is used to estimate weights w (i.e., action type predictions), and w is input into a dynamic action model to dynamically execute different action models: an arm movement model and a grasper operation model. (True value weights) It is obtained based on the state of the robot gripper. When the state changes from open to closed or from closed to open, the robot's motion according to this disclosure is determined by the movement of its arm. Transition to grabber operation The detailed calculation process is explained in the section on dynamic action integration schemes. To leverage the inference capabilities of the VLM, the inference hidden features f from the VLM input to the action model selection module are... rea After processing by a fully connected layer and a normalization layer, the estimated weight w is output.

[0068] The type of action associated with the target object can be determined based on the estimated weight w. For example, if w is less than a first threshold, the action type is determined to be an arm movement; otherwise, the action type is determined to be a gripper operation. The first threshold can be preset, or it can be an empirical or simulated value, for example, it can be set to 0.5.

[0069] Step 304: Select the target action model from the set of candidate action models in the first AI model based on the action type.

[0070] In this embodiment, the candidate action model set includes a variety of pre-trained action models, and each action model corresponds one-to-one with an action type.

[0071] In this embodiment, the dynamic motion model can be a neural network used to predict a series of robot actions, i.e., action blocks. For example, the dynamic motion model may include an arm motion model and a gripper operation model. If the action type is an arm motion, the target motion model is the arm motion model. If the action type is a gripper operation, the target motion model is the gripper operation model. This disclosure proposes a dynamic behavior model mechanism, with step 300 relating to the testing phase and step 400 relating to the training phase.

[0072] Step 305: Based on the second hidden feature and visual features, determine the action sequence associated with the target object through the target action model.

[0073] In this embodiment, as Figure 5 During the testing phase of the dynamic action model shown, the model dynamically executes different action models based on the estimated weights w. If w is less than 0.5, the dynamic action model will use the cognitive hidden features f extracted from the high-level VLM. c and global class marker f cls Given the condition, run the arm motion model to predict a series of multi-step movements. Otherwise, the dynamic motion model will use the cognitive hidden features f extracted from the high-level VLM. c and global class marker f cls Given the conditions, run the grabber operation model to predict a series of multi-step actions.

[0074] In some optional implementations of this embodiment, the candidate action model set includes a first model associated with arm movement and a second model associated with grasper operation; and based on a first hidden feature, the action type associated with the target object is determined by a selection module in the first AI model, including: based on the first hidden feature, outputting a predicted value of the action type associated with the target object by the selection module in the first AI model; determining the action type as arm movement in response to a predicted value less than a first threshold; and determining the action type as grasper operation in response to a predicted value greater than or equal to the first threshold. Selecting one of the two dynamic action models based on the action type to predict the action can improve the accuracy of action prediction.

[0075] In some optional implementations of this embodiment, the visual features include global visual features and local visual features; and based on the second hidden features and the visual features, the action sequence associated with the target object is determined through the target action model, including: in response to the action type being arm movement, determining the action sequence associated with the target object through the first model based on the second hidden features and global visual features; or in response to the action type being a grasper operation, determining the action sequence associated with the target object through the second model based on the second hidden features and local visual features.

[0076] In addition to the cognitive hidden features f extracted from high-level VLM cog In addition to conditions, both action models also input low-level visual features as conditions. This is because the two models focus on different attentional regions of the image (…). Figure 2 (The explanation is in the text), this disclosure uses different visual features as input. Since the arm motion model requires more global attention, this disclosure uses input class labels f cls As a condition, since the grasper operation model requires more local attention, this disclosure specifies the input register flag f. reg As a condition.

[0077] In some optional implementations of this embodiment, the visual features include global visual features and local visual features; and based on the second hidden features and the visual features, the action sequence associated with the target object is determined through a target action model, including: in response to the action type being arm movement, determining the action sequence associated with the target object through a first model based on random noise, the second hidden features and global visual features; or in response to the action type being a grasper operation, determining the action sequence associated with the target object through a second model based on random noise, the second hidden features and local visual features.

[0078] In practical applications, the input to the action model may also include random noise n. rand Dynamic motion models can remove noise and output predicted values ​​for motion sequences.

[0079] In some optional implementations of this embodiment, the feature extraction network includes an encoder, a visual model, and a large language model; and based on the scene image and the first instruction, the feature extraction network in the first AI model determines visual features, a first hidden feature, and a second hidden feature, including: determining language features based on the first instruction through the encoder; determining the first visual feature and the second visual feature based on the scene image through the visual model, wherein the first visual feature is used to characterize the basic structure and local information of the scene image, and the second visual feature is used to characterize the semantic information of the scene image; and determining the first hidden feature and the second hidden feature based on the first visual feature and the language feature through the large language model.

[0080] The first visual feature is used to characterize the basic structure and local information of the image, while the second visual feature is used to characterize the semantic information of the image.

[0081] like Figure 5 As shown, the visual model processes the third-view RGB image input into a set of perceptual labels (i.e., visual features), where the visual features include not only the visual label f vis It also includes the class tag f cls and register flag f reg The visual model can consist of powerful visual transformers DINOv2 and SigLIP, pre-trained on internet-scale image data to capture rich low-level visual features and high-level semantic understanding of observations. At each time step t, the image observation o t The inputs are fed into DINOv2 and SigLIP, respectively generating two downsampled feature maps. and These feature maps are then concatenated along the channel dimension, passed through a linear projection layer, and serialized into a set of visual perceptual labels containing visual labels f. vis =[v1,v2,…,v NV ], class tag f cls and register flag f reg .

[0082] The large language model is responsible for integrating visual information and language instructions, and estimating inference tokens. Here, the LLAMA-2 model can be used as the backbone model. The language instruction l is converted into a set of language tokens f by the LLAMA-2 generator. lan =[l1,l2,…,l NL These tags f lan Then with visual marker f vis This is concatenated with an additional learnable inference tag and processed by a large language model using a causal attention mechanism. The final output is the cognitive hidden feature f. cog (i.e., the second hidden feature) and inference hidden feature frea (i.e., the first hidden feature).

[0083] In some optional implementations of this embodiment, the first hidden feature and the second hidden feature are respectively derived from the hidden features of the second Transformer layer and the last Transformer layer of the LLAMA-2 model. The second Transformer layer is used to select actions, and the last Transformer layer is used to estimate actions.

[0084] f rea and f cog These are the hidden features from the second and last Transformer layers of the large language model, respectively. Cognitive hidden features f cog This serves as a condition for subsequent action model predictions. Inference hidden features f rea As a feature of the subsequent action selection module, it is used to dynamically infer different types of actions.

[0085] In some optional implementations of this embodiment, determining the action sequence associated with the target object based on the second hidden feature and the visual feature through the target action model includes: determining the action sequence associated with the target object based on the second hidden feature and the second visual feature through the target action model.

[0086] like Figure 5 As shown, the action model selection module utilizes the inference capabilities and truth weights (i.e., action type labels) of the VLM. Self-supervised learning is used to estimate weights w (i.e., action type predictions), and w is input into a dynamic action model to dynamically execute different action models: an arm movement model and a grasper operation model. (True value weights) It is obtained based on the state of the robot gripper. When the state changes from open to closed or from closed to open, the robot's motion according to this disclosure is determined by the movement of its arm. Transition to grabber operation The detailed calculation process is explained in the section on dynamic action integration schemes. To leverage the inference capabilities of the VLM, the inference hidden features f from the VLM input to the action model selection module are... rea After processing by a fully connected layer and a normalization layer, the estimated weight w is output.

[0087] In some optional implementations of this embodiment, the feature extraction network includes an LLAMA-2 model. The LLAMA-2 model can serve as the backbone network. LLAMA-2 can fuse speech and visual features to obtain cognitive hidden features f. cogand reasoning hidden features f rea .

[0088] In some optional implementations of this embodiment, the feature extraction network includes a DINOv2 model and a SigLIP model. The visual model in the feature extraction network consists of powerful visual transformers DINOv2 and SigLIP, which have been pre-trained on internet-scale image data to capture rich low-level visual features and high-level semantic understanding of the observations.

[0089] Continue to refer to Figure 4 The diagram illustrates a flow 400 of yet another embodiment of a method performed by an electronic device according to the present disclosure. This method performed by the electronic device includes the following steps:

[0090] Step 401: Obtain training samples.

[0091] In this embodiment, the execution entity of the method for training the robot motion prediction model (e.g.) Figure 7 The server shown can obtain the training sample set in various ways. For example, the executing entity can obtain an existing training sample set stored in a database server via a wired or wireless connection. Another example is that a user can collect training samples using a robot. In this way, the executing entity can receive the training samples collected by the robot and store them locally, thereby generating a training sample set.

[0092] Here, the training sample set may include at least one training sample. The training sample may include images, language instructions, action type labels, and action sequence labels.

[0093] The images here can be third-person perspective images including the robot and the manipulated object. They can be 2D images or stereoscopic images (i.e., images containing depth information). Furthermore, the images can be color images (such as RGB (Red, Green, Blue) photographs) and / or grayscale images, etc. The format of the image is not limited in this disclosure, such as jpg (Joint Photographic Experts Group, a picture format), BMP (Bitmap, an image file format), or RAW (RAW Image Format, a lossless compression format), as long as it can be read and recognized by the executing entity.

[0094] In some embodiments, language instructions can be voice data or data converted from voice data into text.

[0095] In some embodiments, the action type label, also known as the truth value of the action type, may include: arm movement and gripper operation.

[0096] In some embodiments, the action sequence label refers to the robot's actual action space a t =[Δ{x},Δθ,grip].

[0097] Training samples are selected from the training sample set to perform training steps 402 to 405. The method and number of training samples selected are not limited in this disclosure. For example, at least one training sample may be randomly selected, or a training sample with good image clarity (e.g., high pixel count) may be selected.

[0098] Step 402: Based on the image and language instructions, determine the visual features, the first hidden feature and the second hidden feature through a feature extraction network.

[0099] In this embodiment, the feature extraction network can be a neural network used to extract features from language instructions and images. These features may include visual features, a first hidden feature, and a second hidden feature.

[0100] like Figure 5 As shown, the visual model processes the third-view RGB image input into a set of perceptual labels (i.e., visual features), where the visual features include not only the visual label f vis It also includes the class tag f cls and register flag f reg The visual model can consist of powerful visual transformers DINOv2 and SigLIP, pre-trained on internet-scale image data to capture rich low-level visual features and high-level semantic understanding of observations. At each time step t, the image observation o t The inputs are fed into DINOv2 and SigLIP, respectively generating two downsampled feature maps. and These feature maps are then concatenated along the channel dimension, passed through a linear projection layer, and serialized into a set of visual perceptual labels containing visual labels f. vis =[v1,v2,…,v NV ], class tag f cls and register flag f reg .

[0101] The large language model is responsible for integrating visual information and language instructions, and estimating inference tokens. Here, the LLAMA-2 model can be used as the backbone model. The language instruction l is converted into a set of language tokens f by the LLAMA-2 generator. lan =[l1,l2,…,l NL These tags f lan Then with visual marker f visThis is concatenated with an additional learnable inference tag and processed by a large language model using a causal attention mechanism. The final output is the cognitive hidden feature f. cog (i.e., the second hidden feature) and inference hidden feature f rea (i.e., the first hidden feature). f rea and f cog These are the hidden features from the second and last Transformer layers of the large language model, respectively. Cognitive hidden features f cog This serves as a condition for subsequent action model predictions. Inference hidden features f rea As a feature of the subsequent action selection module, it is used to dynamically infer different types of actions.

[0102] Step 403: Based on the first hidden feature, the action type prediction value is determined by the action model selection module, and the first loss is calculated based on the action type prediction value and the action type label.

[0103] In this embodiment, the action model selection module can be a classifier. For example... Figure 5 As shown, the action model selection module utilizes the inference capabilities and truth weights (i.e., action type labels) of the VLM. Self-supervised learning is used to estimate weights w (i.e., action type predictions), and w is input into a dynamic action model to dynamically execute different action models: an arm movement model and a grasper operation model. (True value weights) It is obtained based on the state of the robot gripper. When the state changes from open to closed or from closed to open, the robot's motion according to this disclosure is determined by the movement of its arm. Transition to grabber operation The detailed calculation process is explained in the section on dynamic action integration schemes. To leverage the inference capabilities of the VLM, the inference hidden features f from the VLM input to the action model selection module are... rea After processing by fully connected layers and normalization layers, the estimated weights w are output. The first loss, also known as the class loss L, is calculated based on the predicted action type value and the action type label. class .

[0104] The estimated weight w is supervised by the following cross-entropy loss:

[0105]

[0106] Other loss functions can also be used for the first loss, and no restrictions are imposed here.

[0107] Step 404: Based on the second hidden features and visual features, determine the predicted values ​​of the action sequence through the dynamic action model, and calculate the second loss based on the predicted values ​​of the action sequence and the action sequence labels;

[0108] In this embodiment, the dynamic motion model can be a neural network used to predict a series of robot actions, i.e., action blocks. The dynamic motion model may include an arm motion model and a gripper operation model. This disclosure proposes a dynamic behavior model mechanism, with step 300 relating to the testing phase and step 400 relating to the training phase.

[0109] During the training phase, the dynamic action model uses ground truth weights. To achieve targeted learning of the arm motion model and the gripper operation model. About The design is explained in more detail in the section on dynamic motion integration.

[0110] The dynamic action model outputs predicted action sequences based on second hidden features and visual features. A second loss is calculated based on the difference between the predicted action sequence and the action sequence label. The second loss can employ loss functions such as L2.

[0111] In addition to the cognitive hidden features f extracted from high-level VLM cog In addition to conditions, both action models also input low-level visual features as conditions. This is because the two models focus on different attentional regions of the image (…). Figure 2 (The explanation is in the text), this disclosure uses different visual features as input. Since the arm motion model requires more global attention, this disclosure uses input class labels f cls As a condition, since the grasper operation model requires more local attention, this disclosure specifies the input register flag f. reg As a condition, the losses of the two dynamic action models are supervised separately.

[0112] If the reinforcement learning policy is based on the diffusion model, the calculation of the second loss is as follows:

[0113]

[0114] Among them ||·|| ∑ It's the Mahalanobis distance, which is based on the truth value weights. The error terms are weighted, where These are the noise estimates of the noisy action sequences in the i-th denoising step of the arm motion model and the grasper operation model, respectively. and These correspond to the ground truth values ​​for the arm motion model and the grasper operation model, respectively, and are randomly generated. Furthermore, the inputs to both motion models also include noisy motion 'a'. move ,a mani amove From true noise and truth value actions Calculated. a mani From true noise and truth value actions The calculation yielded the result.

[0115] Motion loss L move and operational losses L mani The weighted sum can be used as the second loss.

[0116] The total loss is the motion loss L. move Operational loss L mani and category loss L class Weighted sum:

[0117] L=w1*L move +w2*L mani +w3*L class

[0118] w1, w2, and w3 represent hyperparameter weights, which can be set to 1.0, 1.0, and 0.0001, respectively.

[0119] Step 405: Adjust the network parameters of the robot motion prediction model based on the first loss and the second loss.

[0120] In this embodiment, the total loss is calculated based on the first loss and the second loss. If multiple training samples are selected, the executing entity can determine that the robot motion prediction model training is complete when the total loss of each training sample reaches the target value. Alternatively, the executing entity can count the proportion of samples whose total loss reaches the target value out of the selected samples. When this proportion reaches a preset sample proportion (e.g., 95%), the robot motion prediction model training can be determined to be complete.

[0121] In this embodiment, if the executing entity determines that the robot motion prediction model has been trained successfully, then step 300 can be implemented. If the executing entity determines that the robot motion prediction model has not been trained successfully, then the relevant parameters in the feature extraction network, motion model selection module, and dynamic motion model of the robot motion prediction model can be adjusted. For example, backpropagation technology can be used to modify the weights in each convolutional layer of the robot motion prediction model. The system can also return to step 401 to reselect training samples from the training sample set. This allows the execution of the aforementioned training steps 402-405 to continue.

[0122] It should be noted that the method of selecting training samples is not limited in this disclosure. For example, if there are a large number of training samples in the training sample set, the executing entity can select training samples that have not been selected before.

[0123] If the executing entity determines that the robot motion prediction model has been trained, then the robot motion prediction model can be applied to testing.

[0124] Optionally, the executing entity can store the generated robot motion prediction model locally or send it to the robot or a database server.

[0125] In some optional implementations of this embodiment, the action type label is constructed as follows: the trajectory weight is obtained by combining Gaussian distributions; the action block weight is obtained by exponentially decreasing; and the product of the trajectory weight and the action block weight is used as the action type label.

[0126] To more robustly and accurately distinguish between the arm motion model and the grasper operation model, this disclosure presents a dynamic motion ensemble scheme. Its main objective is to consider the process of the network learning different actions from both global and local perspectives. The global perspective considers the entire trajectory for each task. The local perspective considers each predicted action block. More specifically, for trajectory weights… This disclosure uses a combined Gaussian distribution.

[0127] In some optional implementations of this embodiment, the combined Gaussian distribution includes the left portion of a first Gaussian distribution and the right portion of a second Gaussian distribution, both having the same mean but different variances, wherein the variance of the left portion is greater than the variance of the right portion. For example... Figure 6 As shown, they have the same mean u, which is determined by the time point at which the gripper opens and closes, but their variances differ. Unlike other experiments, in this invention, σ is more important because the gripper manipulation model is more critical before the gripper switching occurs. l It can be set to 9, σ r It can be set to 3. Gaussian distribution. Take the left part, Gaussian distribution Take the right-hand side. Then, adjust the weights for the action blocks. This disclosure presents an exponentially decreasing method. i represents the index of the action block. Combining these two aspects, this disclosure yields the final truth weight. like Figure 6 As shown. Additionally... Through Calculated, if If it is greater than 0.5, then It equals 1, otherwise it equals 0.

[0128] In some optional implementations of this embodiment, the feature extraction network includes an encoder, a visual model, and a large language model; and the feature extraction network determines visual features, a first hidden feature, and a second hidden feature based on images and language instructions, including: determining language features based on language instructions through the encoder; determining the first visual feature and the second visual feature based on images through the visual model; and determining the first hidden feature and the second hidden feature based on the first visual feature and the language feature.

[0129] The first visual feature is a high-level feature used by the large language model, while the second visual feature is a low-level feature used by the dynamic action model. The first visual feature is used to represent the basic structure and local information of the image, while the second visual feature is used to represent the semantic information of the image.

[0130] The large language model can fuse primary visual features and linguistic features, extracting primary and secondary hidden features from the intermediate layer. The primary hidden features are used to select which dynamic action model to use. The secondary hidden features are used to predict the robot's actions.

[0131] In some optional implementations of this embodiment, the predicted value of the action sequence is determined based on the second hidden feature and the visual feature through a dynamic action model, including: the predicted value of the action sequence is output by the dynamic action model based on the second hidden feature and the second visual feature.

[0132] The dynamic motion model fuses the second hidden feature and the second visual feature to predict the robot's actions.

[0133] In some optional implementations of this embodiment, the second visual feature includes global visual features and local visual features, the dynamic motion model includes an arm motion model and a grasper operation model; and the motion sequence prediction value is determined based on the second hidden feature and the second visual feature through the dynamic motion model, including: determining the arm motion prediction value based on the second hidden feature and the global visual feature through the arm motion model; and inputting the second hidden feature and the local visual feature into the grasper operation model to determine the grasper motion prediction value.

[0134] Because arm motion models require more global attention, this publication discloses the input class label f. cls As a condition, since the grabber operation model requires more local attention, this disclosure inputs the registration tag f. reg As a condition.

[0135] In some optional implementations of this embodiment, calculating the second loss based on the predicted values ​​of the action sequence and the action sequence labels includes: calculating a motion loss based on the difference between the predicted values ​​of the arm action and the action sequence labels; calculating an operational loss based on the difference between the predicted values ​​of the gripper action and the action sequence labels; and calculating a weighted sum of the motion loss and the operational loss as the second loss.

[0136] By directly using action sequence labels as supervision signals, the arm action prediction value and the grasper action prediction value are trained in a supervised manner to minimize the loss, that is, to make the predicted action as close as possible to the actual action.

[0137] In some optional implementations of this embodiment, the second visual feature includes global visual features and local visual features, the dynamic motion model includes an arm motion model and a grasper operation model; and the motion sequence prediction value is determined based on the second hidden feature and the second visual feature through the dynamic motion model, including: determining the noisy arm motion prediction value based on the predetermined noise, the second hidden feature and the global visual feature through the arm motion model; or determining the noisy grasper motion prediction value based on the noise, the second hidden feature and the local visual features through the grasper operation model.

[0138] Gaussian noise is progressively added to the action sequence, eventually resulting in a completely noisy state. A neural network (e.g., U-Net) is trained to learn how to progressively denoise the noisy state and recover the original action sequence. When executing the policy, starting from the noisy state, the trained network is used to progressively denoise the sequence, generating the final action sequence.

[0139] In some optional implementations of this embodiment, calculating the second loss based on the predicted value of the action sequence and the action sequence label includes: superimposing noise and the action sequence label to generate a noise action label; calculating the noise motion loss based on the difference between the predicted value of the noise arm action and the noise action label; calculating the noise operation loss based on the difference between the predicted value of the noise gripper action and the noise action label; and calculating the weighted sum of the noise motion loss and the noise operation loss as the second loss.

[0140] With Gaussian noise added, the losses of the two dynamic action models are supervised separately, as shown below:

[0141]

[0142]

[0143] Among them ||·|| ∑ It's the Mahalanobis distance, which is based on the truth value weights. The error terms are weighted, where These are the noise estimates of the noisy action sequences in the i-th denoising step of the arm motion model and the grasper operation model, respectively. and These correspond to the ground truth values ​​for the arm motion model and the grasper operation model, respectively, and are randomly generated. Furthermore, the inputs to both motion models also include noisy motion 'a'. move ,a mani a move From true noise and truth value actions Calculated. a mani From true noise and truth value actions The calculation yielded the result.

[0144] Motion loss L move and operational losses L mani The weighted sum can be used as the second loss.

[0145] In some optional implementations of this embodiment, the feature extraction network includes an LLAMA-2 model. The LLAMA-2 model can serve as the backbone network. LLAMA-2 can fuse speech and visual features to obtain cognitive hidden features f. cog and reasoning hidden features f rea .

[0146] In some optional implementations of this embodiment, the feature extraction network includes a DINOv2 model and a SigLIP model. The visual model in the feature extraction network consists of powerful visual transformers DINOv2 and SigLIP, which have been pre-trained on internet-scale image data to capture rich low-level visual features and high-level semantic understanding of the observations.

[0147] In some optional implementations of this embodiment, the first loss is cross-entropy loss, and the second loss is L2 loss. Since the action model selection module performs classification, cross-entropy loss is more effective. The dynamic action model solves a regression problem, therefore L2 loss is more effective.

[0148] The method provided in the above embodiments of this disclosure proposes an action model selection module. It utilizes the inherent inference capability of the Visual Model (VLM) to learn weights, thereby dynamically inferring different action types. A dynamic action model is also proposed. It combines high-level inference information from the VLM with low-level visual detail information into the action model. More importantly, this disclosure designs different visual labels for different action models: an arm motion model and a grasper operation model. For example, the arm motion model focuses more on global visual features, while the grasper operation model focuses more on local visual features.

[0149] See also Figure 7 The diagram illustrates the structure of a robot system according to this disclosure. The robot system includes a robot 701 and a server 702, which communicate with each other via a wireless network. The robot 701 can process a received first instruction locally or send it to the server 702 for processing. The server 702 is configured to execute the method described in process 300 or 400, outputting an action sequence and sending it to the robot 701; the robot 701 is configured to receive the action sequence and perform actions according to the action sequence.

[0150] This disclosure also provides an electronic device including a processor, and optionally, a transceiver and / or memory coupled to the processor, the processor being configured to perform the steps of the method provided in any optional embodiment of this disclosure.

[0151] Figure 8 The diagram shows a structural schematic of an electronic device to which an embodiment of the present invention applies, such as... Figure 8 As shown, Figure 8 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this disclosure. Optionally, the electronic device may be a first network node, a second network node, or a third network node.

[0152] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0153] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0154] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0155] The memory 4003 is used to store computer programs that execute embodiments of the present disclosure, and is controlled by the processor 4001 to execute them. The processor 4001 is used to execute the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0156] This disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0157] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0158] The terms “first,” “second,” “third,” “fourth,” “1,” “2,” etc. (if present) in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in a sequence other than that shown in the figures or text.

[0159] It should be understood that although arrows indicate various operation steps in the flowcharts of the embodiments of this disclosure, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of the embodiments of this disclosure, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured as required, and the embodiments of this disclosure do not limit this.

[0160] The above text and accompanying drawings are provided as examples only to help the reader understand this disclosure. They are not intended and should not be construed as limiting the scope of this disclosure in any way. Although certain embodiments and examples have been provided, it will be apparent to those skilled in the art, based on the content disclosed herein, that changes can be made to the illustrated embodiments and examples, and other similar implementations based on the technical concept of this disclosure can be adopted without departing from the scope of this disclosure, and these modifications and modifications are also within the protection scope of the embodiments of this disclosure.

Claims

1. A method performed by an electronic device, comprising: Receive the first instruction for moving the target object; Based on the scene image and the first instruction, visual features, the first hidden feature and the second hidden feature are determined through the feature extraction network in the first AI model; Based on the first hidden feature, the action type associated with the target object is determined by the selection module in the first AI model; Based on the action type, a target action model is selected from the set of candidate action models in the first AI model; Based on the second hidden feature and the visual feature, the action sequence associated with the target object is determined through the target action model.

2. The method according to claim 1, wherein, The candidate action model set includes a first model associated with arm movements and a second model associated with gripper operations.

3. The method according to claim 2, wherein, The step of determining the action type associated with the target object based on the first hidden feature and through the selection module in the first AI model includes: Based on the first hidden feature, the selection module in the first AI model outputs a predicted value of the action type associated with the target object; In response to the predicted value being less than a first threshold, the action type is determined to be an arm movement; In response to the predicted value being greater than or equal to a first threshold, the action type is determined to be a grabber operation.

4. The method according to claim 2, wherein, The visual features include global visual features and local visual features; and The step of determining the action sequence associated with the target object based on the second hidden feature and the visual feature, through the target action model, includes: In response to the action type being arm movement, the action sequence associated with the target object is determined using the first model based on the second hidden feature and the global visual feature; or In response to the action type being a grabber operation, the action sequence associated with the target object is determined through the second model based on the second hidden feature and the local visual feature.

5. The method according to claim 2, wherein, The visual features include global visual features and local visual features; and The step of determining the action sequence associated with the target object based on the second hidden feature and the visual feature, through the target action model, includes: In response to the action type being arm movement, the action sequence associated with the target object is determined using the first model based on random noise, the second hidden feature, and the global visual features; or In response to the action type being a grabber operation, the action sequence associated with the target object is determined by the second model based on random noise, the second hidden feature, and the local visual features.

6. The method according to claim 1, wherein, The feature extraction network includes an encoder, a visual model, and a large language model; and The step of determining visual features, a first hidden feature, and a second hidden feature based on the scene image and the first instruction, through the feature extraction network in the first AI model, includes: Based on the first instruction, the encoder determines the language features; Based on the scene image, a first visual feature and a second visual feature are determined using the visual model. The first visual feature is used to characterize the basic structure and / or local information of the scene image, and the second visual feature is used to characterize the semantic information of the scene image. Based on the first visual feature and the language feature, the first hidden feature and the second hidden feature are determined through the large language model.

7. The method according to claim 6, wherein, The large language model is an LLAMA-2 model. The first hidden feature and the second hidden feature are respectively derived from the hidden features of the second Transformer layer and the last Transformer layer of the LLAMA-2 model. The second Transformer layer is used to select actions, and the last Transformer layer is used to estimate actions.

8. The method according to claim 6, wherein, The step of determining the action sequence associated with the target object based on the second hidden feature and the visual feature, through the target action model, includes: Based on the second hidden feature and the second visual feature, the action sequence associated with the target object is determined through the target action model.

9. The method according to any one of claims 1-8, wherein, The feature extraction network includes the DINOv2 model and the SigLIP model.

10. A robot system, comprising: The server is configured to perform the method of any one of claims 1-9, outputting a sequence of actions and sending it to the robot; The robot is configured to receive the action sequence and perform actions according to the action sequence.

11. An electronic device, comprising: One or more processors; Storage device, on which one or more computer programs are stored, When the one or more computer programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

Images