A method, apparatus, electronic device, and storage medium for processing embodied robots.

Abstract

This application provides a processing method, apparatus, electronic device, and storage medium for embodied robots, including: acquiring the current visual information, task instruction information, and current robotic arm state information of the embodied robot in a target scene; inputting the current visual information, task instruction information, and current robotic arm state information into a global state intermediate supervision network model for visual prediction processing and motion prediction processing, predicting the next moment's visual information and the robot's action block sequence; after the embodied robot completes the action block sequence, inputting the next moment's visual information and the robotic arm state after completing the action block sequence into the global state intermediate supervision network model to continue visual prediction processing and motion prediction processing, until the embodied robot completes the target task corresponding to the task instruction information and stops prediction. This significantly improves the adaptability and execution accuracy of embodied robots in complex operation scenarios.

Description

Technical Field

[0001] This application relates to the field of intelligent decision-making technology, and in particular to a processing method, apparatus, electronic device and storage medium for an embodied robot. Background Technology

[0002] With the explosive growth of artificial intelligence (AI) technology, embodied intelligence, as a typical method and technology of the AI ​​era, has ushered in tremendous development opportunities. Embodied intelligence refers to the method of combining intelligent systems with physical entities, enabling physical entities to perceive their environment, make decisions, and perform corresponding actions. Therefore, the physical entities integrated with embodied intelligence are also called intelligent agents. "Embodied" is not only about abstract algorithms and data, but also about interacting with the world through physical forms. The application scenarios of embodied intelligence are also extremely broad, integrating intelligent agents into various vertical fields such as intelligent manufacturing and service industries, such as industrial inspection and domestic services, allowing embodied intelligence to lead a new upgrade of manufacturing, service, and other industries. In existing technologies, image data is collected in real time during the execution of the current task by the embodied robot. The image data is then input into a pre-trained improved YOLO model to obtain the inference information features of the target object. However, existing technologies only use a vision module, which leads to the technical problem of low execution accuracy of embodied intelligent robots. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a processing method, device, electronic device and storage medium for embodied robots. By utilizing a global state intermediate supervision network model, the visual information and action block sequence of the next moment can be directly determined until the task is successfully completed, which significantly improves the adaptability and execution accuracy of embodied robots in complex operation scenarios.

[0004] This application provides a method for processing embodied robots, the method comprising:

[0005] Acquire the current visual information, task instruction information, and current robotic arm status information of the embodied robot in relation to the target scene;

[0006] The current visual information, task instruction information, and current robotic arm state information are input into a pre-trained global state intermediate supervision network model for visual prediction and motion prediction processing, thereby predicting the next moment's visual information of the robot and the sequence of motion blocks of the robot.

[0007] After the embodied robot completes the sequence of action blocks, it inputs the visual information of the next moment and the state of the robotic arm after completing the sequence of action blocks into the global state intermediate supervision network model to continue visual prediction processing and action prediction processing until the embodied robot completes the target task corresponding to the task instruction information and stops prediction.

[0008] In one possible implementation, the step of inputting the current visual information, task instruction information, and current robotic arm state information into a pre-trained global state intermediate supervision network model for visual prediction processing and motion prediction processing, to predict the next moment's visual information of the robot and the robot's motion block sequence, includes:

[0009] Based on the global state intermediate supervision network model, visual prediction processing is performed on the current visual information and task instruction information to predict the visual information at the next moment; wherein, the current visual information is composed of multiple sub-visual images;

[0010] Based on the global state intermediate supervision network model, motion prediction processing is performed on the current visual information, the task instruction information, and the current state of the robotic arm to predict the motion block sequence.

[0011] In one possible implementation, the step of performing visual prediction processing on the current visual information and task instruction information based on the global state intermediate supervision network model to predict the visual information at the next moment includes:

[0012] In the global state intermediate supervision network model, the sub-visual images acquired by the left robotic arm, right robotic arm and head of the android are subjected to depth image feature extraction, bidirectional cross-attention fusion processing and gated loop fusion processing, and the first visual feature corresponding to each sub-visual image is output.

[0013] In the global state intermediate supervision network model, the task instruction information, multiple sub-visual images, and the first visual features corresponding to each sub-visual image are processed by multimodal segmentation and full attention mechanism to output the visual information of the next time step.

[0014] In one possible implementation, for the sub-visual images of the left robotic arm, the output of the first visual feature corresponding to each sub-visual image includes:

[0015] The features extracted from the sub-visual image are fused by the feature extraction network based on the global state intermediate supervision network model and the visual feature extraction network to obtain RGB visual features;

[0016] A residual deep visual feature extraction network layer composed of multiple convolutional layers is used to extract features from the RGB visual features to generate multiple low-level features of the RGB visual features. A deep feature extraction network is used to extract deep features from the multiple low-level features to obtain the deep features corresponding to each low-level feature.

[0017] Each of the underlying features and its corresponding deep features is subjected to bidirectional cross-attention processing to obtain multiple first fusion features;

[0018] Pyramid pooling is performed on each of the underlying features and the corresponding deep features to generate multiple sub-features with different receptive fields. Gated loop fusion is then performed on each of the underlying features and the multiple sub-features of the corresponding deep features to obtain multiple second fused features.

[0019] Multiple first fusion features are upsampled and stitched together to generate two-dimensional visual features, and multiple second fusion features are sampled and stitched together to generate depth visual features.

[0020] The two-dimensional visual features and the depth visual features are subjected to dimensionality transformation processing to generate the first visual feature corresponding to the sub-visual image.

[0021] In one possible implementation, the task instruction information, multiple sub-visual images, and the first visual features corresponding to each sub-visual image are processed using multimodal segmentation and full attention mechanisms within the global state intermediate supervision network model to output the next-time visual information, including:

[0022] Text feature extraction is performed on the task instruction information;

[0023] For any of the sub-visual images, the task instruction information and the sub-visual image are subjected to multimodal segmentation processing to determine the segmented image. The features in the segmented image are subjected to cross-attention processing with the first visual features of the sub-visual image to obtain the second visual features.

[0024] The transformer network layer is used to process multiple first visual features, multiple second visual features, and text features using a full attention mechanism, and outputs the visual information at the next time step.

[0025] In one possible implementation, the motion prediction processing based on the current visual information, the task instruction information, and the current robotic arm state in the global state intermediate supervision network model to predict the motion block sequence includes:

[0026] Feature extraction is performed on the current state of the robotic arm to determine multiple current robotic arm motion features;

[0027] The action block sequence is predicted by applying a causal attention mechanism to multiple first visual features, multiple second visual features, and multiple current robotic arm motion features based on the transformer network layer.

[0028] In one possible implementation, the global state intermediate supervision network model is determined through the following steps:

[0029] The sample visual information, sample task instruction information, and sample robotic arm state information of the embodied robot are input into the initial global state intermediate supervision network model for visual prediction processing and action prediction processing, so as to predict the visual information of the embodied robot at the next moment and the predicted action block sequence of the embodied robot.

[0030] Based on the actual next-moment visual information, actual action block sequence, predicted next-moment visual information, and cross-entropy loss value between the predicted action block sequence of the embodied robot, the network parameters of the initial global state intermediate supervision network model are iteratively trained to determine the global state intermediate supervision network model.

[0031] This application embodiment also provides a processing device for an embodied robot, the processing device comprising:

[0032] The acquisition module is used to acquire the current visual information, task instruction information, and current robotic arm status information of the embodied robot in relation to the target scene;

[0033] The first determining module is used to input the current visual information, task instruction information, and current robotic arm state information into a pre-trained global state intermediate supervision network model for visual prediction processing and motion prediction processing, and to predict the next moment's visual information of the robot and the sequence of motion blocks of the robot.

[0034] The second determining module is used to input the visual information of the next moment and the state of the robotic arm after the embodied robot has executed the action block sequence into the global state intermediate supervision network model to continue visual prediction processing and action prediction processing until the embodied robot completes the target task corresponding to the task instruction information and stops prediction.

[0035] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the embodied robot processing method described above are performed.

[0036] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the embodied robot processing method described above.

[0037] This application provides a processing method, apparatus, electronic device, and storage medium for an embodied robot. The processing method includes: acquiring the current visual information, task instruction information, and current robotic arm state information of the embodied robot in a target scene; inputting the current visual information, task instruction information, and current robotic arm state information into a pre-trained global state intermediate supervision network model for visual prediction processing and motion prediction processing to predict the next moment's visual information and the robot's action block sequence; after the embodied robot completes the action block sequence, inputting the next moment's visual information and the robotic arm state after completing the action block sequence into the global state intermediate supervision network model to continue visual prediction processing and motion prediction processing until the embodied robot completes the target task corresponding to the task instruction information and stops prediction. Using the global state intermediate supervision network model, the next moment's visual information and action block sequence can be directly determined until the task is successfully completed, significantly improving the adaptability and execution accuracy of the embodied robot in complex operation scenarios.

[0038] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A flowchart illustrating a processing method for an embodied robot provided in an embodiment of this application;

[0041] Figure 2One of the structural schematic diagrams of a processing device for an embodied robot provided in an embodiment of this application;

[0042] Figure 3 A second schematic diagram of the structure of a processing device for an embodied robot provided in an embodiment of this application;

[0043] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0045] First, the applicable application scenarios of this application will be introduced. This application can be applied to the field of intelligent decision-making technology.

[0046] Research has revealed that with the explosive iteration of artificial intelligence (AI) technology, embodied intelligence, as a typical method and technology of the AI ​​era, has ushered in a tremendous development opportunity. Embodied intelligence refers to the method of combining intelligent systems with physical entities, enabling physical entities to perceive their environment, make decisions, and perform corresponding actions. Therefore, the physical entities integrated with embodied intelligence are also called intelligent agents. "Embodied" is not only about abstract algorithms and data, but also about interacting with the world through physical forms. The application scenarios of embodied intelligence are also extremely broad, integrating intelligent agents into various vertical fields such as intelligent manufacturing and service industries, such as industrial inspection and domestic services, allowing embodied intelligence to lead a new upgrade of manufacturing, service, and other industries. In existing technologies, image data is collected in real time during the execution of the current task by the embodied robot, and the image data is input into a pre-trained improved YOLO model to obtain the inference information features of the target object. However, existing technologies only use a vision module, which leads to the technical problem of low execution accuracy of embodied intelligent robots.

[0047] Based on this, embodiments of this application provide a processing method, device, electronic device, and storage medium for embodied robots. By utilizing a global state intermediate supervision network model, the visual information and action block sequence of the next moment can be directly determined until the task is successfully completed, which significantly improves the adaptability and execution accuracy of embodied robots in complex operation scenarios.

[0048] Please see Figure 1 , Figure 1 This is a flowchart illustrating a processing method for an embodied robot provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the processing method includes:

[0049] S101: Acquire the current visual information, task instruction information, and current robotic arm status information of the embodied robot in relation to the target scene.

[0050] In this step, the robot acquires its current visual information, task instruction information, and current robotic arm status information in relation to the target scene.

[0051] Here, the current visual information includes multiple sub-visual images, which are acquired by different robotic arms of the robot. The RGB images acquired by the camera on the left robotic arm, the RGB image acquired by the right robotic arm, and the RGB image acquired by the head at the current moment are respectively denoted as follows: , as well as And record their depths in order as follows: , as well as .

[0052] S102: Input the current visual information, task instruction information, and current robotic arm state information into a pre-trained global state intermediate supervision network model for visual prediction processing and motion prediction processing, and predict the next moment's visual information of the robot and the sequence of motion blocks of the robot.

[0053] In this step, the current visual information, task instruction information, and current robotic arm status information are input into the global state intermediate supervision network model for visual prediction processing and motion prediction processing, predicting the next moment's visual information of the robot and the sequence of motion blocks of the robot.

[0054] It's important to note that intermediate supervision refers to introducing additional supervisory signals (such as loss functions) into the intermediate layers of a neural network to guide the model in learning more robust feature representations and avoid problems like vanishing gradients and training instability found in deep networks. Global state intermediate supervision models build upon intermediate supervision models by introducing the visual state at the next time step (t+1) as a supervisory signal. This means the model not only predicts actions but also the visual changes following those actions.

[0055] In one possible implementation, the step of inputting the current visual information, task instruction information, and current robotic arm state information into a pre-trained global state intermediate supervision network model for visual prediction processing and motion prediction processing, to predict the next moment's visual information of the robot and the robot's motion block sequence, includes:

[0056] A: Based on the global state intermediate supervision network model, visual prediction processing is performed on the current visual information and task instruction information to predict the visual information at the next moment; wherein, the current visual information is composed of multiple sub-visual images.

[0057] Here, the visual information at the next moment is predicted by performing visual prediction processing on the current visual information and task instruction information based on the global state intermediate supervision network model.

[0058] The next-moment visual information includes the next-moment visual information of the left robotic arm, the next-moment visual information of the right robotic arm, and the next-moment visual information of the robot's head.

[0059] In one possible implementation, the step of performing visual prediction processing on the current visual information and task instruction information based on the global state intermediate supervision network model to predict the visual information at the next moment includes:

[0060] (1): In the global state intermediate supervision network model, the sub-visual images acquired by the left robotic arm, right robotic arm and head of the embodied robot are subjected to depth image feature extraction, bidirectional cross attention fusion processing and gated loop fusion processing, and the first visual feature corresponding to each sub-visual image is output.

[0061] Here, in the global state intermediate supervision network model, the sub-visual images acquired by the left robotic arm, right robotic arm and head of the embodied robot are subjected to depth image feature extraction, bidirectional cross-attention fusion processing and gated loop fusion processing, and the first visual feature corresponding to each sub-visual image is output.

[0062] In one possible implementation, for the sub-visual images of the left robotic arm, the output of the first visual feature corresponding to each sub-visual image includes:

[0063] a: The features extracted from the sub-visual image are fused by the feature extraction network based on the global state intermediate supervision network model and the visual feature extraction network to obtain RGB visual features.

[0064] Here, a feature extraction network (DINOv2 feature extraction network) is used to extract features from the sub-visual image, and a visual feature extraction network (visual Transformer network layer) is used to extract contextual features from the sub-visual image. Then, the features extracted by the two feature extraction network layers are fused to obtain RGB visual features.

[0065] It's worth noting that the Visual Transformer network divides an image into fixed-size "image patches," flattens these patches, and uses them as input sequences. It then processes these patches using the Transformer architecture, modeling long-range dependencies through a self-attention mechanism. VIT introduces this mechanism to the vision domain, modeling image features from a global perspective to capture global contextual information within the image.

[0066] b: Based on a residual deep visual feature extraction network layer composed of multiple convolutional layers, feature extraction is performed on the RGB visual features to generate multiple low-level features of the RGB visual features. Based on the deep feature extraction network, deep feature extraction is performed on the multiple low-level features to obtain the deep feature corresponding to each low-level feature.

[0067] Here, the residual deep visual feature extraction network layer includes multiple convolutional modules in sequence, namely convolutional Block_x1, convolutional Block_x2, convolutional Block_x3, convolutional Block_x4, and convolutional Block_x5. The RGB visual features are processed in convolutional Block_x2 to convolutional Block_x5 layers to generate four different scale low-level feature maps, denoted as R1, R2, R3, and R4.

[0068] Each low-level feature map is connected to a Channel Attention Module (CAM) to enhance key semantic channels. For the low-level feature maps, a dedicated Depth extraction network symmetrical to the RGB branch is used (the first layer convolution is replaced with a 1x1 convolution to adapt to single-channel input), outputting four depth feature maps of corresponding scales (denoted as D1, D2, D3, D4).

[0069] c: Perform bidirectional cross-attention processing on each of the underlying features and the corresponding deep features to obtain multiple first fusion features.

[0070] Here, bidirectional cross-attention processing is performed on each low-level feature and its corresponding deep feature to obtain multiple first fusion features.

[0071] Specifically, for the low-level features (R1 and D1, R2 and D2, R3 and D3, R4 and D4), a bidirectional cross-attention mechanism is designed: RGB-guided Depth enhancement. For R2 and D2, R2 is compressed through a 1x1 convolution, then multiplied with D2 to generate a spatial attention map, and D2 is weighted to obtain D2'. Depth-guided RGB enhancement: D2 is compressed through a 1x1 convolution, then multiplied with R2 to generate a spatial attention map, and R2 is weighted to obtain R2'. Feature interaction fusion: R2' and D2' are subjected to "element-wise multiplication + channel concatenation", and then fused into the first fused feature through 3x3 grouped convolution (number of groups = 2). The above steps are performed sequentially for each low-level feature and its corresponding deep feature to determine the corresponding first fused feature.

[0072] d: Perform pyramid pooling processing on each of the underlying features and the corresponding deep features to generate multiple sub-features with different receptive fields, and perform gated loop fusion processing on the multiple sub-features of each of the underlying features and the corresponding deep features to obtain multiple second fused features.

[0073] Here, pyramid pooling is performed on each bottom feature and its corresponding deep feature to generate multiple sub-features with different receptive fields. Gated loop fusion is then performed on each bottom feature and its corresponding deep feature to obtain multiple second fused features.

[0074] The design incorporates a multi-scale gated recurrent fusion unit: Scale decomposition: R2 and D2 are decomposed into four sub-features with different receptive fields (1x1, 3x3, 5x5, 7x7) using pyramid pooling; Gated interaction: For each scale's sub-feature pair (R3_k, D3_k), gating coefficients are generated using a gating mechanism Gk=σ(WrR3_k +WdD3_k)+b), where σ is the sigmoid function and Wr and Wd are learnable parameters; Intra-scale fusion: F3_k=Gk• R3_k +(1-Gk)• D3_k; Inter-scale fusion: The four scales' F3_k are upsampled to a uniform size and then concatenated, followed by 1x1 convolutional dimensionality reduction to obtain F3. The gating coefficient Gk is introduced to estimate modal reliability. By calculating the feature entropy of R3_k and D3_k in the current sample online (the lower the entropy value, the more reliable the feature), the initial gating bias b is dynamically adjusted to give reliable modes higher initial weights. For R3 and D3, R4 and D4, the second fusion features F4, F5 and F6 are obtained in the same way.

[0075] e: Upsample and stitch multiple first fusion features to generate two-dimensional visual features; sample and stitch multiple second fusion features to generate depth visual features; perform dimensionality transformation on the two-dimensional visual features and the depth visual features to generate the first visual features corresponding to the sub-visual image.

[0076] Here, multiple first fusion features are upsampled and stitched together to generate two-dimensional visual features, and multiple second fusion features are sampled and stitched together to generate depth visual features; the two-dimensional visual features and depth visual features are then subjected to dimensionality transformation to generate the first visual features corresponding to the sub-visual image.

[0077] Specifically, features F2 and F3 are upsampled and merged with feature F1 to obtain feature FTE. Simultaneously, features F5 and F6 are upsampled and merged with F4 to obtain feature FTM. Then, based on a cross-attention mechanism, depth visual features and 2D visual features are fused to obtain depth visual features. Dimension transformation is performed on the 2D visual features and depth visual features to generate the first visual feature corresponding to the sub-visual image.

[0078] (2) In the global state intermediate supervision network model, the task instruction information, multiple sub-visual images and the first visual features corresponding to each sub-visual image are processed by multimodal segmentation and full attention mechanism to output the visual information at the next moment.

[0079] Here, in the global state intermediate supervision network model, multimodal segmentation and full attention mechanisms are applied to the task instruction information, multiple sub-visual images, and the first visual features corresponding to each sub-visual image to output the visual information of the next time step.

[0080] In one possible implementation, the task instruction information, multiple sub-visual images, and the first visual features corresponding to each sub-visual image are processed using multimodal segmentation and full attention mechanisms within the global state intermediate supervision network model to output the next-time visual information, including:

[0081] I: Extract text features from the task instruction information.

[0082] II: For any of the sub-visual images, the task instruction information and the sub-visual image are subjected to multimodal segmentation processing to determine the segmented image. The features in the segmented image are subjected to cross-attention processing with the first visual features of the sub-visual image to obtain the second visual features.

[0083] Here, the task instructions for robot operation need to be input into the T5 network layer (Text-to-TextTransfer Transformer) of the model to obtain text features. For the left robotic arm, the text instructions and the corresponding RGB two-dimensional image captured by the camera are input into the model. The object to be executed in the task is found in the two-dimensional image and subjected to reasonable semantic segmentation. The foreground of the segmented image retains RGB information, and the background information is covered with RGB pixels of (0,0,0). The image is then fed into the convolution module for feature extraction. The obtained features are cross-attention with the first visual features from the left robotic arm camera to finally obtain the second visual features.

[0084] III: Based on the transformer network layer, the first visual features, the second visual features, and the text features are processed by a full attention mechanism to output the visual information at the next time step.

[0085] Here, the next-time visual information is output by processing multiple first visual features, multiple second visual features, and text features using a full attention mechanism based on the transformer network layer.

[0086] B: Based on the current visual information, the task instruction information, and the current state of the robotic arm, the action block sequence is predicted by performing motion prediction processing on the global state intermediate supervision network model.

[0087] Here, the action block sequence is predicted by performing motion prediction processing on the current visual information, task instruction information, and the state of the front robotic arm in the global state intermediate supervision network model.

[0088] The current state of the robotic arm includes the movement state of the left robotic arm and the movement state of the right robotic arm.

[0089] In one possible implementation, the motion prediction processing based on the current visual information, the task instruction information, and the current robotic arm state in the global state intermediate supervision network model to predict the motion block sequence includes:

[0090] Feature extraction is performed on the current state of the robotic arm to determine multiple current robotic arm motion features; based on the transformer network layer, causal attention mechanism is applied to multiple first visual features, multiple second visual features, and multiple current robotic arm motion features to predict the action block sequence.

[0091] Here, features are extracted from the current state of the robotic arm to determine multiple current robotic arm motion features; based on the transformer network layer, multiple first visual features, multiple second visual features, and multiple current robotic arm motion features are processed by a causal attention mechanism to predict the action block sequence.

[0092] In one possible implementation, the global state intermediate supervision network model is determined through the following steps:

[0093] The sample visual information, sample task instruction information, and sample robotic arm state information of the avatar robot are input into the initial global state intermediate supervision network model for visual prediction and action prediction processing, predicting the next-time visual information and the predicted action block sequence of the avatar robot. Based on the actual next-time visual information, actual action block sequence, predicted next-time visual information, and cross-entropy loss value between the predicted action block sequence, the network parameters of the initial global state intermediate supervision network model are iteratively trained to determine the global state intermediate supervision network model.

[0094] Here, to handle different robotic arms of different robots during training, additional tokens are introduced. These tokens are obtained by merging the current robotic arm pose (embedding) and the current robotic arm motion frequency (embedding) and then processing them through an MLP. Simultaneously, a PAD token of corresponding size is added later to introduce intermediate supervision information. Finally, a Tokenaction is added to predict the final execution action. The aforementioned tokens are then continuously input into the transformer network to predict action tokens. In the process of visual information prediction, the PAD token replaces other visual tokens. This is done to introduce feedback supervision information, enabling the model to predict what visual changes will occur after it performs its predicted action. This allows the model to better understand the direction of action execution and improve the accuracy of action prediction. It is necessary to predict the visual information generated by the left robotic arm at time +1 after performing the predicted action. Similarly, it is necessary to predict the visual information generated by the head and right arm at time +1. This information is represented by tokens. Then, the visual tokens are decoded based on RQ-VAE, and the MSE loss is calculated by comparing them with the real visual images. This design not only makes the model sensitive to the influence of its own predictions, but also allows visual information from different locations to learn from each other, improving the understanding of spatial location and thus increasing the accuracy of model predictions. To effectively perform visual predictions for the next time step, the visual information at time t+1 can be mutually attenuated based on the visual token at time t; however, the visual information at time t cannot see the visual information at time t+1. Similarly, for action prediction, a causal attention mechanism is still used for action tokens. Finally, the final action token dimension is discretized, and each continuous action dimension is mapped to 256 discrete bins, minimizing the cross-entropy loss of action prediction during training.

[0095] S103: After the embodied robot completes the action block sequence, it inputs the visual information of the next moment and the state of the robotic arm after completing the action block sequence into the global state intermediate supervision network model to continue visual prediction processing and action prediction processing until the embodied robot completes the target task corresponding to the task instruction information and stops prediction.

[0096] In this step, after the embodied robot completes the action block sequence, the visual information of the next moment and the state of the robotic arm after completing the action block sequence are input into the global state intermediate supervision network model to continue visual prediction processing and action prediction processing until the embodied robot completes the target task corresponding to the task instruction information and stops prediction.

[0097] It should be noted that the global state intermediate supervision network model consists of a visual processing network layer and an action processing network layer. The action processing network layer is composed of transformer network layers. The visual processing network layer includes a feature extraction network, a visual feature extraction network, a residual deep visual feature extraction network layer consisting of multiple convolutional layers, a channel attention module, a multi-scale gated recurrent fusion unit, a T5 network layer (Text-to-Text Transfer Transformer), convolutional modules, and transformer network layers. The feature extraction network and the visual feature extraction network are two parallel extraction networks. Following the feature extraction network and the visual feature extraction network, the residual deep visual feature extraction network layer consisting of multiple convolutional layers, the channel attention module, and the multi-scale gated recurrent fusion unit are sequentially connected. The T5 network layer (Text-to-Text Transfer Transformer) in the visual processing network layer is connected to the convolutional modules, which in turn are connected to the transformer network layer and the multi-scale gated recurrent fusion unit.

[0098] This application provides a processing method for an embodied robot, characterized by the following steps: acquiring the embodied robot's current visual information, task instruction information, and current robotic arm state information for a target scene; inputting the current visual information, task instruction information, and current robotic arm state information into a pre-trained global state intermediate supervision network model for visual prediction and action prediction processing, predicting the embodied robot's next-moment visual information and action block sequence; after the embodied robot completes the action block sequence, inputting the next-moment visual information and the robotic arm state after completing the action block sequence into the global state intermediate supervision network model for continued visual prediction and action prediction processing, until the embodied robot completes the target task corresponding to the task instruction information, at which point prediction stops. Utilizing the global state intermediate supervision network model, the next-moment visual information and action block sequence can be directly determined until the task is successfully completed, significantly improving the adaptability and execution accuracy of the embodied robot in complex operation scenarios.

[0099] Please see Figure 2 , Figure 3 , Figure 2 One of the structural schematic diagrams of a processing device for an embodied robot provided in an embodiment of this application; Figure 3 This is a second schematic diagram of the structure of a processing device for an embodied robot provided in an embodiment of this application. For example... Figure 2 As shown, the processing device 200 of the embodied robot includes:

[0100] The acquisition module 210 is used to acquire the current visual information, task instruction information and current robotic arm status information of the embodied robot in relation to the target scene;

[0101] The first determining module 220 is used to input the current visual information, task instruction information, and current robotic arm state information into a pre-trained global state intermediate supervision network model for visual prediction processing and motion prediction processing, and to predict the next moment's visual information of the robot and the sequence of motion blocks of the robot.

[0102] The second determining module 230 is used to input the visual information of the next moment and the state of the robotic arm after the embodied robot has completed the action block sequence into the global state intermediate supervision network model to continue visual prediction processing and action prediction processing until the embodied robot completes the target task corresponding to the task instruction information and stops prediction.

[0103] Furthermore, the first determining module 220 is used to input the current visual information, task instruction information, and current robotic arm state information into a pre-trained global state intermediate supervision network model for visual prediction processing and action prediction processing, predicting the next moment's visual information of the robot and the robot's action block sequence:

[0104] Based on the global state intermediate supervision network model, visual prediction processing is performed on the current visual information and task instruction information to predict the visual information at the next moment; wherein, the current visual information is composed of multiple sub-visual images;

[0105] Based on the global state intermediate supervision network model, motion prediction processing is performed on the current visual information, the task instruction information, and the current state of the robotic arm to predict the motion block sequence.

[0106] Furthermore, the first determining module 220 is used to perform visual prediction processing on the current visual information and task instruction information based on the global state intermediate supervision network model, and predict the visual information at the next moment:

[0107] In the global state intermediate supervision network model, the sub-visual images acquired by the left robotic arm, right robotic arm and head of the android are subjected to depth image feature extraction, bidirectional cross-attention fusion processing and gated loop fusion processing, and the first visual feature corresponding to each sub-visual image is output.

[0108] In the global state intermediate supervision network model, the task instruction information, multiple sub-visual images, and the first visual features corresponding to each sub-visual image are processed by multimodal segmentation and full attention mechanism to output the visual information of the next time step.

[0109] Furthermore, the first determining module 220 is used for the sub-visual images of the left robotic arm, and outputs the first visual feature corresponding to each sub-visual image:

[0110] The features extracted from the sub-visual image are fused by the feature extraction network based on the global state intermediate supervision network model and the visual feature extraction network to obtain RGB visual features;

[0111] A residual deep visual feature extraction network layer composed of multiple convolutional layers is used to extract features from the RGB visual features to generate multiple low-level features of the RGB visual features. A deep feature extraction network is used to extract deep features from the multiple low-level features to obtain the deep features corresponding to each low-level feature.

[0112] Each of the underlying features and its corresponding deep features is subjected to bidirectional cross-attention processing to obtain multiple first fusion features;

[0113] Pyramid pooling is performed on each of the underlying features and the corresponding deep features to generate multiple sub-features with different receptive fields. Gated loop fusion is then performed on each of the underlying features and the multiple sub-features of the corresponding deep features to obtain multiple second fused features.

[0114] Multiple first fusion features are upsampled and stitched together to generate two-dimensional visual features, and multiple second fusion features are sampled and stitched together to generate depth visual features.

[0115] The two-dimensional visual features and the depth visual features are subjected to dimensionality transformation processing to generate the first visual feature corresponding to the sub-visual image.

[0116] Furthermore, the first determining module 220 is used in the global state intermediate supervision network model to perform multimodal segmentation processing and full attention mechanism processing on the task instruction information, multiple sub-visual images, and the first visual features corresponding to each sub-visual image, and outputs the visual information at the next time step:

[0117] Text feature extraction is performed on the task instruction information;

[0118] For any of the sub-visual images, the task instruction information and the sub-visual image are subjected to multimodal segmentation processing to determine the segmented image. The features in the segmented image are subjected to cross-attention processing with the first visual features of the sub-visual image to obtain the second visual features.

[0119] The transformer network layer is used to process multiple first visual features, multiple second visual features, and text features using a full attention mechanism, and outputs the visual information at the next time step.

[0120] Furthermore, the first determining module 220 is used to perform motion prediction processing on the current visual information, the task instruction information, and the current state of the robotic arm based on the global state intermediate supervision network model, and predict the motion block sequence:

[0121] Feature extraction is performed on the current state of the robotic arm to determine multiple current robotic arm motion features;

[0122] The action block sequence is predicted by applying a causal attention mechanism to multiple first visual features, multiple second visual features, and multiple current robotic arm motion features based on the transformer network layer.

[0123] Furthermore, such as Figure 3 As shown, the processing device 200 of the embodied robot also includes a model training module 240, which determines the global state intermediate supervision network model through the following steps:

[0124] The sample visual information, sample task instruction information, and sample robotic arm state information of the embodied robot are input into the initial global state intermediate supervision network model for visual prediction processing and action prediction processing, so as to predict the visual information of the embodied robot at the next moment and the predicted action block sequence of the embodied robot.

[0125] Based on the actual next-moment visual information, actual action block sequence, predicted next-moment visual information, and cross-entropy loss value between the predicted action block sequence of the embodied robot, the network parameters of the initial global state intermediate supervision network model are iteratively trained to determine the global state intermediate supervision network model.

[0126] This application provides a processing device for an embodied robot, comprising: an acquisition module for acquiring current visual information, task instruction information, and current robotic arm state information of the embodied robot in a target scene; a first determination module for inputting the current visual information, task instruction information, and current robotic arm state information into a pre-trained global state intermediate supervision network model for visual prediction processing and action prediction processing, predicting the next moment's visual information and the robot's action block sequence; and a second determination module for inputting the next moment's visual information and the robotic arm state after the embodied robot completes the action block sequence into the global state intermediate supervision network model to continue visual prediction processing and action prediction processing until the embodied robot completes the target task corresponding to the task instruction information. By utilizing the global state intermediate supervision network model, the next moment's visual information and action block sequence can be directly determined until the task is successfully completed, significantly improving the adaptability and execution accuracy of the embodied robot in complex operation scenarios.

[0127] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0128] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, they can perform the operations described above. Figure 1 The specific implementation of the processing method of the embodied robot in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0129] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The specific implementation of the processing method of the embodied robot in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0130] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0131] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0132] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0133] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0134] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing embodied robots, characterized in that, The processing method includes: Acquire the current visual information, task instruction information, and current robotic arm status information of the embodied robot in relation to the target scene; The current visual information, task instruction information, and current robotic arm state information are input into a pre-trained global state intermediate supervision network model for visual prediction and motion prediction processing. This predicts the next-moment visual information of the robot and the robot's motion block sequence, including: Based on the global state intermediate supervision network model, visual prediction processing is performed on the current visual information and task instruction information to predict the visual information at the next moment; wherein, the current visual information is composed of multiple sub-visual images; Based on the global state intermediate supervision network model, the current visual information, the task instruction information, and the current state of the robotic arm are processed to predict the action block sequence. The step of performing visual prediction processing on the current visual information and task instruction information based on the global state intermediate supervision network model to predict the visual information at the next moment includes: In the global state intermediate supervision network model, the sub-visual images acquired by the left robotic arm, right robotic arm and head of the android are subjected to depth image feature extraction, bidirectional cross-attention fusion processing and gated loop fusion processing, and the first visual feature corresponding to each sub-visual image is output. In the global state intermediate supervision network model, the task instruction information, multiple sub-visual images, and the first visual features corresponding to each sub-visual image are processed by multimodal segmentation and full attention mechanism to output the visual information of the next time step. After the embodied robot completes the sequence of action blocks, it inputs the visual information of the next moment and the state of the robotic arm after completing the sequence of action blocks into the global state intermediate supervision network model to continue visual prediction processing and action prediction processing until the embodied robot completes the target task corresponding to the task instruction information and stops prediction.

2. The processing method according to claim 1, characterized in that, For the sub-visual images of the left robotic arm, the output of the first visual feature corresponding to each sub-visual image includes: The features extracted from the sub-visual image are fused by the feature extraction network based on the global state intermediate supervision network model and the visual feature extraction network to obtain RGB visual features; A residual deep visual feature extraction network layer composed of multiple convolutional layers is used to extract features from the RGB visual features to generate multiple low-level features of the RGB visual features. A deep feature extraction network is used to extract deep features from the multiple low-level features to obtain the deep features corresponding to each low-level feature. Each of the underlying features and its corresponding deep features is subjected to bidirectional cross-attention processing to obtain multiple first fusion features; Pyramid pooling is performed on each of the underlying features and the corresponding deep features to generate multiple sub-features with different receptive fields. Gated loop fusion is then performed on each of the underlying features and the multiple sub-features of the corresponding deep features to obtain multiple second fused features. Multiple first fusion features are upsampled and stitched together to generate two-dimensional visual features, and multiple second fusion features are sampled and stitched together to generate depth visual features. The two-dimensional visual features and the depth visual features are subjected to dimensionality transformation processing to generate the first visual feature corresponding to the sub-visual image.

3. The processing method according to claim 1, characterized in that, The global state intermediate supervision network model performs multimodal segmentation and full attention processing on the task instruction information, multiple sub-visual images, and the first visual features corresponding to each sub-visual image, and outputs the visual information for the next time step, including: Text feature extraction is performed on the task instruction information; For any of the sub-visual images, the task instruction information and the sub-visual image are subjected to multimodal segmentation processing to determine the segmented image. The features in the segmented image are subjected to cross-attention processing with the first visual features of the sub-visual image to obtain the second visual features. The transformer network layer is used to process multiple first visual features, multiple second visual features, and text features using a full attention mechanism, and outputs the visual information at the next time step.

4. The processing method according to claim 1, characterized in that, The motion prediction process, based on the current visual information, the task instruction information, and the current state of the robotic arm, is performed in the global state intermediate supervision network model to predict the motion block sequence, including: Feature extraction is performed on the current state of the robotic arm to determine multiple current robotic arm motion features; The action block sequence is predicted by applying a causal attention mechanism to multiple first visual features, multiple second visual features, and multiple current robotic arm motion features based on the transformer network layer.

5. The processing method according to claim 1, characterized in that, The global state intermediate supervision network model is determined through the following steps: The sample visual information, sample task instruction information, and sample robotic arm state information of the embodied robot are input into the initial global state intermediate supervision network model for visual prediction processing and action prediction processing, so as to predict the visual information of the embodied robot at the next moment and the predicted action block sequence of the embodied robot. Based on the actual next-moment visual information, actual action block sequence, predicted next-moment visual information, and cross-entropy loss value between the predicted action block sequence of the embodied robot, the network parameters of the initial global state intermediate supervision network model are iteratively trained to determine the global state intermediate supervision network model.

6. A processing device for an embodied robot, characterized in that, The processing device is applied to the processing method of the android according to any one of claims 1-5, and the processing device comprises: The acquisition module is used to acquire the current visual information, task instruction information, and current robotic arm status information of the embodied robot in relation to the target scene; The first determining module is used to input the current visual information, task instruction information, and current robotic arm state information into a pre-trained global state intermediate supervision network model for visual prediction processing and motion prediction processing, and to predict the next moment's visual information of the robot and the sequence of motion blocks of the robot. The second determining module is used to input the visual information of the next moment and the state of the robotic arm after the embodied robot has executed the action block sequence into the global state intermediate supervision network model to continue visual prediction processing and action prediction processing until the embodied robot completes the target task corresponding to the task instruction information and stops prediction.

7. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the processing method of the embodied robot as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the processing method for the embodied robot as described in any one of claims 1 to 5.

Images