Method for determining motion sequence model of embodied robot, method for determining motion sequence and related products

Abstract

Embodiments of the present disclosure provide a method for determining a motion sequence model based on a body robot, a method for determining a motion sequence and related products, which relate to the technical field of body robots, and the method comprises: obtaining training samples and motion sequence labels; processing a plurality of image frame pairs and task instructions using a motion sequence model with frozen parameters to obtain visual text feature information and key frame weights, and determining reference state feature information based on the key frame weights and state frames; processing the visual text feature information and the reference state feature information using a to-be-trained adaptation module to obtain a reference motion sequence; in the training process of the motion sequence model, adjusting the parameters of the to-be-trained adaptation module based on a loss function value determined based on the motion sequence labels, the reference motion sequence and the key frame weights, and keeping the backbone parameters of the motion sequence model unchanged until the trained motion sequence model meets a training stop condition. In this way, the prediction accuracy of the motion sequence is improved.

Description

Technical Field

[0001] This disclosure relates to the field of embodied robot technology, and in particular to a method for determining motion sequence models based on embodied robots, a method for determining motion sequences, and related products. Background Technology

[0002] With the rapid development of artificial intelligence technology, motion sequence prediction has become a core technology for embodied robots to achieve autonomous decision-making and task execution. Its goal is to generate a sequence of action instructions that can be executed within a future period of time based on environmental perception and task objectives. However, the accuracy of determining the motion sequence of embodied robots in existing technologies is relatively low. Summary of the Invention

[0003] This disclosure provides a method for determining motion sequence models based on androids, a method for determining motion sequences, and related products, in order to at least solve the above-mentioned technical problems existing in the prior art.

[0004] In a first aspect, embodiments of this disclosure provide a method for determining a motion sequence model based on an android, the method comprising:

[0005] Acquire training samples and corresponding action sequence labels. The action sequence labels represent the actual action sequence of the embodied robot corresponding to the training samples. The training samples include image frame pairs of the embodied robot under multiple different perspectives of the target scene, state frames associated with multiple image frame pairs, and task instructions. Each image frame pair includes RGB image frames and depth image frames.

[0006] The action sequence model with parameter freezing is used to process multiple image frame pairs and task instructions to obtain visual text feature information and key frame weights. The key frame weights are used to measure the importance of state frames, and reference state feature information is determined based on the key frame weights and state frames.

[0007] The visual text feature information and reference state feature information are processed by the adaptation module to be trained to obtain the reference action sequence of the embodied robot.

[0008] During the training of the action sequence model, the parameters of the adaptation module to be trained are adjusted based on the loss function value determined by the action sequence label, reference action sequence, and keyframe weights, while keeping the backbone parameters of the action sequence model unchanged, until the trained action sequence model meets the training stopping condition.

[0009] Secondly, embodiments of this disclosure provide a method for determining an action sequence, the method comprising:

[0010] The system acquires actual image frame pairs from multiple different perspectives of the embodied robot in a real-world scenario, actual state frames associated with multiple actual image frame pairs, and actual task instructions. Each actual image frame pair includes an actual RGB image frame and an actual depth image frame.

[0011] Multiple actual image frame pairs, actual state frames, and actual task instructions are input into the action sequence model obtained by the action sequence model determination method based on the embodied robot provided in the first aspect. The action sequence model is then used to process the multiple actual image frame pairs, actual state frames, and actual task instructions to obtain the predicted action sequence of the embodied robot.

[0012] Thirdly, embodiments of this disclosure provide a motion sequence model determination device based on an animate robot, the device comprising:

[0013] The acquisition module is used to acquire training samples and corresponding action sequence labels. The action sequence labels represent the real action sequence of the embodied robot corresponding to the training samples. The training samples include image frame pairs of the embodied robot in multiple different perspectives of the target scene, state frames associated with multiple image frame pairs, and task instructions. Each image frame pair includes RGB image frames and depth image frames.

[0014] The processing module is used to process multiple image frame pairs and task instructions using the parameter-frozen action sequence model to obtain visual text feature information and key frame weights. The key frame weights are used to measure the importance of state frames and determine reference state feature information based on the key frame weights and state frames.

[0015] The processing module is also used to process the visual text feature information and reference state feature information using the adaptation module to be trained, so as to obtain the reference action sequence of the embodied robot.

[0016] The adjustment module is used to adjust the parameters of the adaptation module to be trained during the training process of the action sequence model based on the loss function value determined by the action sequence label, reference action sequence, and keyframe weights, while keeping the backbone parameters of the action sequence model unchanged until the trained action sequence model meets the training stopping condition.

[0017] Fourthly, embodiments of this disclosure provide an action sequence determination apparatus, the apparatus comprising:

[0018] The acquisition module is used to acquire actual image frame pairs of the embodied robot from multiple different perspectives in the actual scene, actual state frames associated with multiple actual image frame pairs, and actual task instructions. Each actual image frame pair includes an actual RGB image frame and an actual depth image frame.

[0019] The processing module is used to input multiple actual image frame pairs, actual state frames, and actual task instructions into the action sequence model obtained by the action sequence model determination method based on the embodied robot provided in the first aspect, and to process the multiple actual image frame pairs, actual state frames, and actual task instructions through the action sequence model to obtain the predicted action sequence of the embodied robot.

[0020] Fifthly, embodiments of this disclosure provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the motion sequence model determination method based on the embodied robot provided in the first aspect or the motion sequence determination method provided in the second aspect.

[0021] In a sixth aspect, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions, which are used to cause a computer to execute the motion sequence model determination method based on an embodied robot provided in the first aspect or the motion sequence determination method provided in the second aspect.

[0022] In a seventh aspect, embodiments of this disclosure provide a computer program product, which includes a computer program that, when executed by a processor, implements the motion sequence model determination method based on an embodied robot provided in the first aspect or the motion sequence determination method provided in the second aspect.

[0023] Based on the action sequence model determination method for embodied robots provided in this disclosure, since the image frame pairs in the acquired training samples can include RGB image frames and depth image frames, the visual understanding ability of the action sequence model can be effectively enhanced during model training. Furthermore, by using the parameter-frozen action sequence model to process multiple image frame pairs and task instructions in the training samples to obtain visual text feature information and keyframe weights, and determining reference state feature information based on these keyframe weights and state frames, the introduction of keyframe weights enhances the model's sensitivity to keyframe recognition, thereby improving the accuracy of fine-grained operation prediction. Next, the visual text feature information and reference state feature information can be processed using the adaptation module to be trained to obtain a reference action sequence for the embodied robot. Then, during training, based on the loss function value determined by the action sequence label, reference action sequence, and keyframe weights, only the parameters of the adaptation module to be trained are adjusted, while keeping the core parameters of the action sequence model unchanged until the training stopping condition is met. This achieves the adoption of a partial parameter fine-tuning strategy, significantly optimizing the prediction effect of downstream tasks while maintaining the model's generalization ability, especially in scenarios with few samples, effectively improving inference accuracy. This can effectively improve the accuracy of motion sequence models, thereby enhancing the prediction accuracy of motion sequences for embodied robots in practical applications.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0025] Figure 1 A flowchart illustrating a method for determining a motion sequence model based on an embodied robot, provided in an embodiment of this disclosure;

[0026] Figure 2 A flowchart illustrating an action sequence determination method provided in this embodiment of the disclosure;

[0027] Figure 3 This is a schematic diagram of the structure of a motion sequence model determination device based on an embodiment of a hymenoidae robot provided in this disclosure;

[0028] Figure 4 This is a schematic diagram of the structure of an action sequence determination device provided in an embodiment of this disclosure;

[0029] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0030] To make the objectives, features, and advantages of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0031] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0034] This disclosure provides a method for determining motion sequence models based on androids, a method for determining motion sequences, and related products, in order to at least solve the technical problem of low motion sequence prediction accuracy in the prior art.

[0035] It should be noted that the execution entity of the motion sequence model determination method based on android provided in this disclosure can be a motion sequence model determination device based on android, or a part of the motion sequence model determination device used to execute the motion sequence model determination method based on android. This disclosure describes in detail the motion sequence model determination method based on android provided in this disclosure by taking the execution of the motion sequence model determination method based on android by a motion sequence model determination device as an example.

[0036] Accordingly, the action sequence determination method provided in this disclosure can be executed by an action sequence determination device or a module within that device that performs the action sequence determination method. This disclosure describes the action sequence determination method provided in this disclosure in detail, using the example of an action sequence determination device executing the action sequence determination method.

[0037] The method for determining the motion sequence model based on a android provided in this disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] Figure 1 This is a flowchart illustrating a method for determining a motion sequence model based on an embodied robot, as provided in an embodiment of this disclosure.

[0039] like Figure 1 As shown, the execution entity of the method for determining the motion sequence model based on a tangible robot can be a device for determining the motion sequence model based on a tangible robot. Based on this, the method for determining the motion sequence model based on a tangible robot provided in this embodiment of the present disclosure may specifically include the following steps:

[0040] S110, obtain the training samples and the action sequence labels corresponding to the training samples.

[0041] Before introducing this step, it should be noted that the action sequence model involved in this embodiment requires multiple iterations to adjust its loss function value until the loss function value of the trained action sequence model meets the training stopping condition, thus obtaining a trained action sequence model. If only one training sample is input during each iteration, the sample size is too small to effectively train and adjust the action sequence model. Therefore, multiple training samples are needed to iterate the action sequence model. Thus, the training samples in this embodiment can be multiple, and no specific limitation is made here.

[0042] In some embodiments, the action sequence labels corresponding to the training samples can be used to characterize the actual action sequences of the embodied robot corresponding to the training samples. Accordingly, the training samples may include multiple pairs of image frames from different perspectives of the embodied robot in relation to a target scene, state frames associated with these multiple image frame pairs, and task instructions. Each pair of image frames includes an RGB image frame and a depth image frame from the same perspective. For example, these multiple image frame pairs may include a left arm image frame pair from the left arm's perspective, a right arm image frame pair from the right arm's perspective, and a head image frame pair from the embodied robot's global head perspective. The left arm image frame pair may include a left arm RGB image frame and a left arm depth image frame; the right arm image frame pair may include a right arm RGB image frame and a right arm depth image frame; and the head image frame pair may include a head RGB image frame and a head depth image frame. No specific limitations are imposed here. The state frames may contain the embodied robot's pose data, and the task instructions may be data used to instruct the embodied robot to perform relevant tasks. No specific limitations are imposed here.

[0043] Specifically, it is possible to acquire multiple training samples and label the action sequence of each training sample manually, or directly label the action sequence of each training sample using a motion sequence model determination device based on the embodied robot, without making any specific limitations here.

[0044] S120 uses a parameter-frozen action sequence model to process multiple image frame pairs and task instructions to obtain visual text feature information and key frame weights, and determines reference state feature information based on key frame weights and state frames.

[0045] The motion sequence model for freezing the above parameters can be a pre-trained model used to predict the motion sequence of the embodied robot; no specific limitations are imposed here.

[0046] In some embodiments, the aforementioned associated frame weights are used to measure the importance of the corresponding state frame of the embodied robot. Specifically, a larger keyframe weight indicates a greater importance of the corresponding state frame, and vice versa. No specific limitation is made here. Therefore, determining reference state feature information based on keyframe weights and state frames can be used to characterize the attention of the state frame.

[0047] Specifically, after obtaining the training samples, the training samples can be input into the parameter-frozen action sequence model. The action sequence model processes multiple image frame pairs in the training samples and the task instructions to obtain visual text feature information and key frame weights. Then, the reference state feature information of the embodied robot can be determined based on the key frame weights and the state frames in the training samples.

[0048] In one example, during the determination of reference state feature information, the state frames in the training samples can be encoded to obtain state encoding, and the keyframe weights can be encoded to obtain keyframe weight encoding. The reference state feature information can then be obtained based on the product of this state encoding and the keyframe weight encoding, thus representing the attention. Furthermore, to obtain more accurate reference state feature information, the state encoding and keyframe weight encoding can be input separately into a fully connected layer, and the state encoding processed by the fully connected layer and the keyframe weight encoding can be multiplied together to obtain the corresponding reference state feature information.

[0049] S130: The visual text feature information and reference state feature information are processed by the adaptation module to be trained to obtain the reference action sequence of the embodied robot.

[0050] Specifically, after obtaining the visual text feature information and reference state feature information corresponding to the training sample, the visual text feature information and reference state feature information corresponding to the training sample can be processed by the training adaptation module to obtain the reference action sequence of the embodied robot.

[0051] S140, during the training of the action sequence model, the parameters of the adaptation module to be trained are adjusted based on the loss function value determined by the action sequence label, reference action sequence, and keyframe weights, while keeping the backbone parameters of the action sequence model unchanged, until the trained action sequence model meets the training stopping condition.

[0052] The training stopping condition can be preset based on actual experience or circumstances. For example, the training stopping condition can be set to the loss function value of the trained action sequence model being less than a preset function value. The preset loss function value can be preset based on actual experience or circumstances, and no specific limitation is made here.

[0053] In this way, during the training process of the action sequence model, based on the loss function value determined by the action sequence label, reference action sequence, and keyframe weights, only the parameters of the adaptation module to be trained are adjusted, while keeping the backbone parameters of the action sequence model unchanged, until the trained action sequence model meets the training stopping condition.

[0054] Based on the action sequence model determination method for embodied robots provided in this disclosure, since the image frame pairs in the acquired training samples can include RGB image frames and depth image frames, the visual understanding ability of the action sequence model can be effectively enhanced during model training. Furthermore, by using the parameter-frozen action sequence model to process multiple image frame pairs and task instructions in the training samples to obtain visual text feature information and keyframe weights, and determining reference state feature information based on these keyframe weights and state frames, the introduction of keyframe weights enhances the model's sensitivity to keyframe recognition, thereby improving the accuracy of fine-grained operation prediction. Next, the visual text feature information and reference state feature information can be processed using the adaptation module to be trained to obtain a reference action sequence for the embodied robot. Then, during training, based on the loss function value determined by the action sequence label, reference action sequence, and keyframe weights, only the parameters of the adaptation module to be trained are adjusted, while keeping the core parameters of the action sequence model unchanged until the training stopping condition is met. This achieves the adoption of a partial parameter fine-tuning strategy, significantly optimizing the prediction effect of downstream tasks while maintaining the model's generalization ability, especially in scenarios with few samples, effectively improving inference accuracy. This can effectively improve the accuracy of motion sequence models, thereby enhancing the prediction accuracy of motion sequences for embodied robots in practical applications.

[0055] To accurately and thoroughly explain the method for determining the motion sequence model based on an embodied robot provided in this disclosure, in one embodiment, the aforementioned training adapter module may include a multimodal adapter and an action adapter. The multimodal adapter may be composed of a serial convolution, an activation function, a convolutional layer, a sigmoid function (Sigmoid), and a deconvolutional layer. The action adapter may be composed of a serial convolution, an activation function, a convolutional layer, a sigmoid function, and a deconvolutional layer. No specific limitations are made here.

[0056] Based on this, the above-mentioned S130 may specifically include the following steps:

[0057] The visual text feature information is processed using a multimodal adapter to obtain the processed visual text feature information, and reference visual text feature information is determined based on the visual text feature information and the processed visual text feature information.

[0058] The motion prediction information of the embodied robot is processed using a motion adapter to obtain processed motion prediction information. Based on the motion prediction information and the processed motion prediction information, the corresponding reference motion sequence of the embodied robot is determined.

[0059] In some embodiments, the motion prediction information of the android can be determined based on reference visual text feature information and reference state feature information, without specific limitations. Furthermore, the reference motion sequence can include multiple motions, each consisting of the joint angles of the android's robotic arm and the corresponding gripper opening and closing states, without specific limitations.

[0060] Specifically, since the aforementioned adaptation module to be trained may include a multimodal adapter and an action adapter, the action sequence model determination device based on the embodied robot can first use the multimodal adapter to process the aforementioned visual text feature information to obtain processed visual text feature information, and then determine reference visual text feature information based on the visual text feature information and the processed visual text feature information. More specifically, the action sequence model determination device based on the embodied robot can obtain reference visual text feature information by adding the visual text feature information and the processed visual text feature information.

[0061] Next, the motion prediction information of the embodied robot, determined based on reference visual text feature information and reference state feature information, can be processed using a motion adapter to obtain processed motion prediction information. Based on this processed motion prediction information, a reference motion sequence for the embodied robot can be determined. More specifically, the motion sequence model determination device for the embodied robot can add the motion sequence prediction information and the processed motion prediction information, and then process the sum of the two using a motion prediction head to obtain the reference motion sequence for the embodied robot. The motion prediction head consists of multiple linear layers, activation layers, and normalization layers; specific limitations are not specified here.

[0062] Furthermore, since the aforementioned action prediction information can be determined based on reference visual text feature information and reference state feature information, it should be noted that in this embodiment, a diffusion model can be used to process the reference visual text feature information and reference state feature information to obtain the action prediction information of the embodied robot. The aforementioned diffusion model can be obtained through iterative training using a flow matching method, which is not specifically limited here.

[0063] In this embodiment, a multimodal adapter can be used to process visual text feature information, and reference visual text feature information can be obtained based on the processed visual text feature information and the reference state feature information. Then, an action adapter can be used to process the action prediction information determined based on the reference visual text feature information and the reference state feature information, and a reference action sequence can be determined based on the processed action prediction information and the action prediction information. In this way, by introducing a multimodal adapter and an action adapter, the prediction performance of downstream tasks can be significantly optimized while maintaining the model's generalization ability during subsequent training.

[0064] To accurately and comprehensively describe the action sequence model determination method based on androids provided in this disclosure, in one embodiment, the action sequence model may include a visual feature extraction module, a text feature extraction module, and a visual-text fusion module. Thus, the above-described S120 may specifically include the following steps:

[0065] The visual feature extraction module is used to process the RGB image frames and depth image frames in multiple image frame pairs to obtain three-dimensional visual features;

[0066] The text feature extraction module is used to process the task instructions to obtain text features;

[0067] The visual-text fusion module is used to process 3D visual features and text features to obtain visual-text feature information and keyframe weights.

[0068] Specifically, since the action sequence model involved in this embodiment may include a visual feature extraction module, a text feature extraction module, and a visual-text fusion module, the visual feature extraction module can be used to process the RGB image frames and depth image frames in multiple image frame pairs to obtain three-dimensional visual features, and the text feature extraction module can be used to process the task instruction to obtain the corresponding text features. Then, the visual-text fusion module can be used to process the three-dimensional visual features and text features to obtain the corresponding visual-text feature information and keyframe weights.

[0069] In this embodiment, by processing RGB image frames and depth feature frames from multiple image frame pairs from different perspectives, three-dimensional visual features can be accurately obtained, effectively improving the model's ability to perceive the environment. Furthermore, task instructions can be processed to obtain corresponding text features, and these three-dimensional visual features and text features are fused to obtain corresponding visual-text feature information and keyframe weights. Thus, accurately obtaining visual-text feature information and keyframe weights not only improves the accuracy of model execution but also allows for the measurement of the importance of state frames.

[0070] To accurately acquire three-dimensional visual features and thus enhance the model's ability to perceive the environment, in one embodiment, the aforementioned visual feature extraction module may include a two-dimensional feature extraction submodule, a depth feature extraction submodule, a three-dimensional feature extraction submodule, and a visual feature fusion submodule.

[0071] Based on this, the steps described above, which utilize the visual feature extraction module to process RGB image frames and depth image frames from multiple image frame pairs to obtain three-dimensional visual features, may specifically include the following steps:

[0072] The two-dimensional feature extraction submodule is used to extract features from RGB image frames in multiple image frame pairs to obtain two-dimensional visual features.

[0073] The depth feature extraction submodule is used to extract features from depth image frames in multiple image frame pairs to obtain depth visual features.

[0074] The 3D feature extraction submodule is used to extract features from the RGBD information corresponding to multiple image frame pairs to obtain initial 3D visual features. The RGBD information corresponding to each image frame pair is determined based on the RGB image frame and the depth image frame in the image frame pair.

[0075] The visual feature fusion submodule is used to fuse two-dimensional visual features, depth visual features, and initial three-dimensional visual features to obtain three-dimensional visual features.

[0076] Specifically, since each image frame pair can include an RGB image frame and a depth image frame, after acquiring multiple image frame pairs from different viewpoints, the 2D feature extraction submodule can be used to extract features from the RGB images in these multiple image frame pairs to obtain 2D visual features. Simultaneously, the depth feature extraction submodule can be used to extract features from the depth image frames in these multiple image frame pairs to obtain depth visual features. Furthermore, the 3D feature extraction submodule can be used to extract features from the RGBD information corresponding to multiple image frame pairs to obtain initial 3D visual features. Based on this, the visual feature fusion submodule can be used to fuse the 2D visual features, depth visual features, and initial 3D visual features to obtain the final 3D visual features.

[0077] In one example, the aforementioned multiple image frame pairs may include left arm image frame pairs from the left arm perspective, right arm image frame pairs from the right arm perspective, and head image frame pairs from the head global perspective. The left arm image frame pair may include a left arm RGB image frame and a left arm depth image frame; the right arm image frame pair may include a right arm RGB image frame and a right arm depth image frame; and the head image frame pair may include a head RGB image frame and a head depth image frame. Based on this, a two-dimensional feature extraction submodule can be used to extract features from the aforementioned left arm RGB image frames, head RGB image frames, and right arm RGB image frames to obtain corresponding two-dimensional visual features. Simultaneously, a depth feature extraction submodule can be used to extract features from the aforementioned left arm depth image frames, head depth image frames, and right arm depth image frames to obtain corresponding depth visual features. Furthermore, a three-dimensional feature extraction submodule can be used to process the left arm RGBD information determined based on the aforementioned left arm RGB image frames and left arm depth image frames, the head RGBD information determined based on the aforementioned head RGB image frames and head depth image frames, and the right arm RGBD information determined based on the aforementioned right arm RGB image frames and right arm depth image frames to obtain initial three-dimensional features. Based on this, the visual feature fusion submodule can be used to fuse the two-dimensional visual features, depth visual features, and initial three-dimensional visual features to obtain the final three-dimensional visual features.

[0078] In this embodiment, accurate two-dimensional visual features can be obtained by processing RGB image frames from multiple different viewpoints. Simultaneously, accurate depth visual features can be obtained by processing depth image frames from multiple different viewpoints. Furthermore, the RGBD information determined from the RGB and depth image frames from multiple viewpoints can be processed to obtain initial three-dimensional visual features. These two-dimensional visual features, depth visual features, and initial three-dimensional visual features can then be fused to obtain the final three-dimensional visual features. This effectively improves the model's ability to perceive its environment and enhances the accuracy of model execution.

[0079] Based on this, in one embodiment, the aforementioned two-dimensional feature extraction submodule may include multiple two-dimensional feature extraction units and two-dimensional feature stitching units. Each two-dimensional feature extraction unit includes at least two different extraction subunits and stitching subunits. Therefore, the step of using the two-dimensional feature extraction submodule to extract features from RGB image frames in multiple image frame pairs to obtain two-dimensional visual features may specifically include the following steps:

[0080] For each image frame pair, at least two different extraction sub-units in the corresponding two-dimensional feature extraction unit are used to extract the RGB image frames in the image frame pair to obtain at least two different RGB image features.

[0081] For each image frame pair, the splicing subunit in the two-dimensional feature extraction unit corresponding to the image frame pair is used to splice at least two different RGB image features to obtain the target image features of the image frame pair.

[0082] Two-dimensional visual features are obtained by stitching together the target image features corresponding to multiple image frame pairs using two-dimensional feature stitching units.

[0083] Specifically, the aforementioned two-dimensional feature extraction submodule can include multiple two-dimensional feature extraction units and two-dimensional feature stitching units. Each two-dimensional feature extraction unit can include at least two different extraction subunits and stitching subunits. Thus, for each image frame pair, at least two different extraction subunits within the corresponding two-dimensional feature extraction subunit can be used to extract RGB image features from the image frame pair, resulting in at least two different RGB image features. Then, the stitching subunit within the two-dimensional feature extraction unit can be used to stitch these at least two different RGB image features together to obtain the target image features for the image frame pair. Therefore, after obtaining the target image features corresponding to multiple image frame pairs, the two-dimensional feature stitching unit can be used to stitch these target image features together to obtain two-dimensional visual features.

[0084] Continuing with the example above, each extraction subunit can consist of a visual model, a transposed convolutional layer, and a fully connected layer. For example, the first extraction subunit of at least two extraction subunits can consist of a Scalable internet-guided Linguistic-Perceptual Pretraining V2 (SigLipV2), a transposed convolutional layer, and a fully connected layer. The second extraction subunit can consist of a Distillation with NO labels v3 (Dinov3) visual model, a transposed convolutional layer, and a fully connected layer. Thus, in the process of extracting features from the left arm RGB image frame, head RGB image frame and right arm RGB image frame using the two-dimensional feature extraction submodule to obtain the corresponding two-dimensional visual features, taking the left arm RGB image as an example, the left arm RGB image frame can be simultaneously input into the SigLipV2 model and the Dinov3 model. The outputs of the SigLipV2 model and the Dinov3 model are respectively sent to the corresponding transposed convolutional layer and fully connected layer to obtain at least two different RGB image features. Then, the stitching subunit can be used to stitch these at least two different RGB image features to obtain the target image features corresponding to the left arm.

[0085] Furthermore, following the above process, the target image features corresponding to the right arm and the head can be obtained sequentially. Thus, the two-dimensional feature stitching subunit can be used to stitch and fuse the target image features corresponding to the left arm, the right arm, and the head to obtain two-dimensional visual features.

[0086] In this embodiment, for each image frame pair, at least two RGB image features are extracted from the RGB image through at least two different extraction subunits in its corresponding two-dimensional feature extraction subunit. Then, these different RGB image features are fused using a stitching subunit to generate the target image features for that image frame pair. After obtaining the target image features for all image frame pairs, these features are further integrated and stitched together by a two-dimensional feature stitching unit to obtain the final two-dimensional visual features. This effectively fuses multi-angle feature information, improving the accuracy and robustness of two-dimensional visual feature representation.

[0087] To accurately acquire depth visual features, in one embodiment, the aforementioned depth feature extraction submodule may include a depth feature compression unit, a deconvolution unit, an unpooling unit, and a depth feature fusion unit. Based on this, the step of using the depth feature extraction submodule to extract features from depth image frames in multiple image frame pairs to obtain depth visual features may specifically include the following steps:

[0088] For each image frame pair, the depth image frame in the image frame pair is compressed using the depth feature compression unit to obtain the first depth image feature;

[0089] The first depth image features are sequentially input into the deconvolution unit and the unpooling unit. The compressed depth image features are then restored through the deconvolution unit and the unpooling unit to obtain the second depth image features.

[0090] The deep feature fusion unit is used to stitch together and fuse the second depth image features corresponding to multiple image frame pairs to obtain depth visual features.

[0091] Specifically, since the depth feature extraction submodule can include a depth feature compression unit, a deconvolution unit, an unpooling unit, and a depth feature fusion unit, for each image frame pair, the depth image frame in the image frame pair can be compressed using the depth feature compression unit to obtain compressed depth image features, i.e., the first depth image features. Then, the first depth image features can be sequentially input into the deconvolution unit and the unpooling unit. The first depth image features are processed by the deconvolution unit and the unpooling unit to obtain the second depth image features. Then, the depth feature fusion unit can be used to stitch and fuse the second depth image features corresponding to multiple image frame pairs to obtain depth visual features.

[0092] In one example, the depth image frames from each image frame pair can be input into a depth feature compression unit, which can be constructed from convolutional sub-units and downsampling sub-units. The convolutional sub-unit can contain a two-dimensional dilated convolutional layer, an activation layer, a pooling layer, and a normalization layer. Thus, after inputting the depth image frames from each image frame into the depth feature compression unit, convolution processing is performed followed by downsampling to obtain the first depth image feature. This first depth feature is then sequentially input into the deconvolutional unit and the depooling unit to obtain the second depth features from different viewpoints. Finally, the depth features from each viewpoint are stitched together and fused to obtain the depth visual feature.

[0093] It should be noted that the number of depth feature compression units is not specifically limited in this embodiment. If there are at least two depth feature compression units, the first depth image features output by the at least two depth feature compression units can be concatenated, and the concatenated first depth image features can be sequentially input into the deconvolution unit and the unpooling unit for subsequent processing. Furthermore, after obtaining the first depth image features using the depth feature compression units, the first depth image features can also be input into a convolution unit, and the features output by the convolution unit can be concatenated and fused before being input into the deconvolution unit and the unpooling unit for subsequent processing.

[0094] In this embodiment, for each image frame pair, the depth feature extraction submodule first compresses the depth image frame using a depth feature compression unit to obtain a first depth image feature. Then, this feature is sequentially input into a deconvolution unit and an unpooling unit for processing to obtain a second depth image feature. After all image frame pairs have undergone the above processing, the depth feature fusion unit concatenates and fuses the second depth image features to ultimately generate a unified depth visual feature. Thus, through multi-layer feature extraction and fusion, the representation capability and accuracy of depth information are effectively improved.

[0095] To accurately obtain the initial 3D visual features, in one embodiment, the aforementioned multiple image frame pairs include robotic arm image frame pairs and head image frame pairs; the aforementioned 3D feature extraction submodule may include an RGBD segmentation unit, a 3D feature extraction unit, and a first multi-head attention unit; based on this, the aforementioned step of using the 3D feature extraction submodule to extract features from the RGBD information corresponding to multiple image frame pairs to obtain the initial 3D visual features may specifically include:

[0096] The RGBD segmentation unit is used to segment the corresponding RGBD information of the robotic arm image frames to obtain the region information of the target object;

[0097] The three-dimensional feature extraction unit is used to extract features from the regional information of the target object to obtain local three-dimensional features;

[0098] The 3D feature extraction unit is used to perform feature processing on the corresponding RGBD information of the head image frames to obtain global 3D features;

[0099] Using global 3D features as the query vector and local 3D features as the key and value vectors, the global and local 3D features are input into the first multi-head attention unit. The first multi-head attention unit processes the global and local 3D features to obtain the initial 3D visual features.

[0100] The target object can be an item or gripper that the robot is trying to grab; there are no specific limitations here.

[0101] Specifically, since the multiple image frame pairs in this embodiment may include robotic arm image frame pairs and head image frame pairs, and the three-dimensional feature extraction submodule in this embodiment may include an RGBD segmentation unit, a three-dimensional feature extraction unit, and a first multi-head attention unit, the RGBD segmentation unit can be used to segment the RGBD information corresponding to the robotic arm image frame pairs to obtain the region information of the target object, and the three-dimensional feature extraction unit can be used to extract features from the region information of the target object to obtain local three-dimensional features. Meanwhile, considering that the head camera needs to acquire global feature information, it is not necessary to input the RGBD information corresponding to the head image frame pairs into the RGBD segmentation unit for segmentation. Instead, the RGBD information corresponding to the head image frame pairs can be directly input into the three-dimensional feature extraction unit. The three-dimensional feature extraction unit can then perform feature processing on the RGBD information corresponding to the head image frame pairs to obtain global three-dimensional features. Furthermore, the global three-dimensional features can be used as the query vector, and the local three-dimensional features as the key and value vectors can be input into the first multi-head attention unit. The first multi-head attention unit processes the global and local three-dimensional features to obtain initial three-dimensional visual features.

[0102] In one example, the RGBD information corresponding to the robotic arm image frames is input into the RGBD segmentation unit. This unit uses a mask to segment the areas of the gripper and the manipulated object, revealing the target object's region. Other areas are filled with padding. Then, a 3D feature extraction unit, constructed by concatenating 3D convolutional layers, fully connected layers, and reshaping layers, extracts local 3D features. Conversely, because the head camera needs to acquire global feature information, it doesn't need to input the RGBD information; the RGBD information corresponding to the head image frames can be directly input into the 3D feature extraction unit to obtain global 3D features. Based on this, the global 3D features can be used as the query vector, and the local 3D features as the key and value vectors, respectively. These global and local 3D features are then input into a first multi-head attention unit. The first multi-head attention unit processes the global and local 3D features to obtain initial 3D visual features, which are then stitched and fused to obtain 3D visual features with strong target location information. The aforementioned robotic arm image frame pairs include left-arm robotic arm image frame pairs and right-arm robotic arm image frames, without specific limitations here.

[0103] In this embodiment, the RGBD segmentation unit can segment the RGBD information corresponding to the robotic arm image frames to extract the region information of the target object. Subsequently, the 3D feature extraction unit extracts features from this region information to obtain local 3D features. Since the head image frames are used to obtain global scene information, their corresponding RGBD information is not segmented but directly input into the 3D feature extraction unit to extract global 3D features. Then, the global 3D features are used as the query vector, and the local 3D features are used as the key and value vectors, respectively, and input into the first multi-head attention unit for feature interaction and fusion, outputting initial 3D visual features. In this way, the complementarity and focusing of local and global features enable the subsequent generation of accurate 3D visual features.

[0104] In one embodiment, the aforementioned visual feature fusion submodule includes a dilated convolutional unit, a first multi-head self-attention unit, a second multi-head attention unit, and a convolutional attention unit. Based on this, the step of fusing two-dimensional visual features, depth visual features, and initial three-dimensional visual features using the visual feature fusion submodule to obtain three-dimensional visual features may specifically include the following steps:

[0105] The first feature is obtained by processing the two-dimensional visual features using dilated convolutional units;

[0106] The initial 3D visual features are processed using the first multi-head self-attention unit to obtain the second feature;

[0107] Using the first feature as the key vector and value vector, and the second feature as the query vector, the first feature and the second feature are input into the second multi-head attention unit. The first feature and the second feature are then fused through the second multi-head attention unit to obtain the third feature.

[0108] The third feature is fused using a convolutional attention unit, along with a fourth feature obtained by multiplying the third feature with the depth visual feature, to obtain the three-dimensional visual feature.

[0109] Specifically, the visual feature fusion submodule of this embodiment includes a dilated convolutional unit, a first multi-head self-attention unit, a second multi-head attention unit, and a convolutional attention unit. Based on this, the dilated convolutional unit can process two-dimensional visual features to obtain a first feature, and the first multi-head self-attention unit can process three-dimensional visual features to obtain a second feature. Then, using the first feature as a key vector and a value vector, and the second feature as a query vector, the first and second features can be input into the second multi-head attention unit. The second multi-head attention unit can then fuse the first and second features to obtain a third feature. To achieve more complete depth information fusion, the convolutional attention unit can fuse the third feature and a fourth feature obtained by multiplying the third feature with the depth visual features to obtain the three-dimensional visual feature. It should also be noted that before inputting the first and second features into the second multi-head self-attention unit, a fully connected layer can be used to process the first and second features to obtain a more accurate result.

[0110] In this embodiment, the dilated convolutional unit can be used to process two-dimensional visual features to obtain a first feature, and the first multi-head self-attention unit can be used to process three-dimensional visual features to obtain a second feature. Then, using the first feature as a key vector and a value vector, and the second feature as a query vector, the first and second features are input into a second multi-head attention unit. The second multi-head attention unit fuses the first and second features to obtain a third feature, and the convolutional attention unit fuses the third feature with a fourth feature obtained by multiplying the third feature with the depth visual features to generate a three-dimensional visual feature with strong representational capabilities. Thus, through multi-level cross-modal feature fusion, the accuracy and completeness of the three-dimensional visual features are effectively improved.

[0111] In one embodiment, the step of processing 3D visual features and text features using the visual-text fusion module to obtain visual-text feature information may specifically include the following steps:

[0112] Using text features as query vectors and 3D visual features as key and value vectors, the text features and 3D visual features are input into the third multi-head attention unit. The text features and 3D visual features are processed by the third multi-head attention unit to obtain the first visual text fusion feature.

[0113] By combining text features and 3D visual features, a second visual text fusion feature is obtained.

[0114] By combining the first visual text fusion feature and the second visual text fusion feature, visual text feature information is obtained.

[0115] Specifically, it can use text features as query vectors and three-dimensional visual features as key and value vectors, input the text features and three-dimensional visual features into a third multi-head attention unit, process the text features and three-dimensional visual features through the third multi-head attention unit to obtain a first visual text fusion feature, and can splice the first visual text fusion feature and the second visual text fusion feature to obtain the final visual text feature information.

[0116] In this embodiment, textual features can be used as the query vector, and 3D visual features can be used as the key and value vectors. These textual and 3D visual features are input into a third multi-head attention unit. The third multi-head attention unit processes the textual and 3D visual features to obtain a first visual-text fusion feature. This first visual-text fusion feature is then concatenated with a second visual-text fusion feature to generate visual-text information that combines semantic alignment and multimodal representation capabilities. Thus, through hierarchical feature fusion, the model's ability to understand and express cross-modal associations is effectively enhanced.

[0117] To provide a comprehensive and detailed description of the action sequence model determination method based on embodied robots provided in this disclosure, in one embodiment, the steps described above—adjusting the parameters of the multimodal adaptation module and the action adaptation module using the loss function value determined based on action sequence labels, reference action sequences, and keyframe weights, while keeping the core parameters of the action sequence model unchanged, until the trained action sequence model meets the training stopping condition—may specifically include the following steps:

[0118] Calculate the initial loss function value between the real action sequence indicated by the action sequence label and the reference action sequence;

[0119] The loss function value is determined by multiplying the initial loss function value by the keyframe weights.

[0120] If the loss function value does not meet the training stopping condition, adjust the parameters of the module to be trained and keep the backbone parameters of the action sequence model unchanged until the trained action sequence model meets the training stopping condition.

[0121] Specifically, the initial loss function value between the real action sequence indicated by the action sequence label and the reference action sequence can be calculated first. Then, the loss function value can be determined based on the product of the initial loss function value and the keyframe weights. In this way, if the loss function value does not meet the training stopping condition, only the parameters of the adaptation module to be trained can be adjusted, while keeping the backbone parameters of the action sequence model unchanged, until the trained action sequence model meets the training stopping condition.

[0122] In this embodiment, keyframe weights are introduced when calculating the loss function value, which enhances the model's sensitivity to keyframe recognition. By prioritizing the optimization of keyframes and weakening non-keyframes, the accuracy of fine-grained operation predictions is improved. Simultaneously, a partial parameter fine-tuning strategy is employed to significantly optimize the prediction performance of downstream tasks while maintaining the model's generalization ability, especially in scenarios with few samples, effectively improving inference accuracy.

[0123] This disclosure also provides a method for determining an action sequence, which is described below in conjunction with the appendix. Figure 2 The action sequence determination method provided in this disclosure will be described in detail through specific embodiments.

[0124] Figure 2 This is a flowchart illustrating an action sequence determination method provided in an embodiment of this disclosure. Figure 2 As shown, the execution subject of this method can be an action sequence determination device. Based on this, the method can specifically include the following steps:

[0125] S210: Acquire actual image frame pairs from multiple different perspectives of the embodied robot in the actual scene, actual state frames associated with multiple actual image frame pairs, and actual task instructions.

[0126] S220, inputs multiple actual image frame pairs, actual status frames, and actual task instructions to the application Figure 1 The action sequence model obtained by the method steps shown is used to process multiple actual image frame pairs, actual state frames and actual task instructions to obtain the predicted action sequence of the embodied robot.

[0127] Each of the aforementioned actual image frame pairs includes an actual RGB image frame and an actual depth image frame, without specific limitations here. Additionally, the aforementioned predicted action sequence may include multiple actions of the embodied robot over a future period, without specific limitations here.

[0128] Specifically, it is possible to acquire actual image frame pairs from multiple different perspectives of the embodied robot in a real scene, actual state frames associated with multiple actual image frame pairs, and actual task instructions. Then, by inputting the actual image frame pairs from multiple different perspectives, the actual state frames associated with multiple actual image frame pairs, and the actual task instructions into the action sequence model obtained by the method steps shown in any of the above embodiments, and by processing the multiple actual image frame pairs, actual state frames, and actual task instructions through the action sequence model, a predicted action sequence of the embodied robot can be obtained.

[0129] It should be noted that after obtaining the predicted motion sequence output by the motion sequence model, the predicted motion sequence can be processed by filtering or data interpolation to ensure the smoothness and continuity of the robotic arm's movement.

[0130] In this embodiment, it is possible to acquire actual image frame pairs from multiple different perspectives of the embodied robot in a real-world scenario, actual state frames associated with these actual image frame pairs, and actual task instructions. The action sequence model obtained using the aforementioned method for determining the action sequence model based on the embodied robot is then used to process these actual image frame pairs, actual state frames, and actual task instructions to obtain the predicted action sequence of the embodied robot. Since the image frame pairs used in the training samples for determining the action sequence model include RGB image frames and depth image frames, the model's visual understanding ability is effectively enhanced. Furthermore, by introducing keyframe weights, the model's sensitivity to keyframe recognition is enhanced, thereby improving the accuracy of fine-grained operation prediction. Simultaneously, a partial parameter fine-tuning strategy is employed to significantly optimize the prediction effect of downstream tasks while maintaining the model's generalization ability, especially in scenarios with few samples, effectively improving inference accuracy. Thus, by determining an action sequence model with high prediction accuracy, the prediction accuracy of action sequences can be effectively improved during the action sequence prediction process.

[0131] Based on the same inventive concept, this disclosure also provides a device for determining a motion sequence model based on a hymenoidae robot. (See attached diagram for details.) Figure 3 The present disclosure provides a detailed description of the motion sequence model determination device based on an embodied robot.

[0132] Figure 3 This is a schematic diagram of a device for determining a motion sequence model based on an embodied robot, provided in an embodiment of this disclosure.

[0133] like Figure 3 As shown, the motion sequence model determination device 300 based on the embodied robot includes:

[0134] The acquisition module 310 is used to acquire training samples and action sequence labels corresponding to the training samples. The action sequence labels represent the real action sequence of the embodied robot corresponding to the training samples. The training samples include image frame pairs of the embodied robot under multiple different perspectives of the target scene, state frames associated with multiple image frame pairs, and task instructions. Each image frame pair includes RGB image frames and depth image frames.

[0135] The processing module 320 is used to process multiple image frame pairs and task instructions using the parameter-frozen action sequence model to obtain visual text feature information and key frame weights. The key frame weights are used to measure the importance of state frames and to determine reference state feature information based on the key frame weights and state frames.

[0136] The processing module 320 is also used to process the visual text feature information and reference state feature information using the adaptation module to be trained, so as to obtain the reference action sequence of the embodied robot.

[0137] The adjustment module 330 is used to adjust the parameters of the adaptation module to be trained during the training process of the action sequence model based on the loss function value determined by the action sequence label, the reference action sequence, and the keyframe weights, while keeping the backbone parameters of the action sequence model unchanged until the trained action sequence model meets the training stopping condition.

[0138] In one embodiment, the adaptation module to be trained includes a multimodal adapter and an action adapter; based on this, the action sequence model determination device based on an embodied robot provided in this disclosure embodiment may include:

[0139] The processing module is specifically used to process visual text feature information using a multimodal adapter to obtain processed visual text feature information, and to determine reference visual text feature information based on the visual text feature information and the processed visual text feature information.

[0140] The processing module is specifically used to process the motion prediction information of the embodied robot using the motion adapter to obtain the processed motion prediction information. Based on the motion prediction information and the processed motion prediction information, the module determines the reference motion sequence of the embodied robot. The motion prediction information is determined based on reference visual text feature information and reference state feature information.

[0141] In one embodiment, the action sequence model includes a visual feature extraction module, a text feature extraction module, and a visual-text fusion module; based on this, the action sequence model determination device based on an embodied robot provided in this disclosure embodiment may include:

[0142] The processing module is specifically used to process the RGB image frames and depth image frames in multiple image frame pairs using the visual feature extraction module to obtain three-dimensional visual features;

[0143] The processing module is specifically used to process task instructions using the text feature extraction module to obtain text features;

[0144] The processing module is specifically used to process 3D visual features and text features using the visual-text fusion module to obtain visual-text feature information and keyframe weights.

[0145] In one embodiment, the visual feature extraction module includes a two-dimensional feature extraction submodule, a depth feature extraction submodule, a three-dimensional feature extraction submodule, and a visual feature fusion submodule; based on this, the motion sequence model determination device based on a hymenoidae robot provided in this disclosure embodiment may include:

[0146] The extraction module is used to extract features from RGB image frames in multiple image frame pairs using the two-dimensional feature extraction submodule to obtain two-dimensional visual features;

[0147] The extraction module is also used to extract features from depth image frames in multiple image frame pairs using the depth feature extraction submodule to obtain depth visual features;

[0148] The extraction module is also used to extract features from the RGBD information corresponding to multiple image frame pairs using the 3D feature extraction submodule to obtain initial 3D visual features. The RGBD information corresponding to each image frame pair is determined based on the RGB image frame and depth image frame in the image frame pair.

[0149] The fusion module is used to fuse two-dimensional visual features, depth visual features, and initial three-dimensional visual features using the visual feature fusion submodule to obtain three-dimensional visual features.

[0150] In one embodiment, the multiple image frame pairs include robotic arm image frame pairs and head image frame pairs; the three-dimensional feature extraction submodule includes an RGBD segmentation unit, a three-dimensional feature extraction unit, and a first multi-head attention unit; based on this, the action sequence model determination device based on a hymenoided robot provided in this disclosure embodiment may include:

[0151] The segmentation module is used to segment the corresponding RGBD information of the robotic arm image frame using the RGBD segmentation unit to obtain the region information of the target object;

[0152] The extraction module is specifically used to extract features from the regional information of the target object using the three-dimensional feature extraction unit to obtain local three-dimensional features;

[0153] The extraction module is specifically used to perform feature extraction processing on the corresponding RGBD information of the head image frame using the three-dimensional feature extraction unit to obtain global three-dimensional features;

[0154] The input module is used to input the global 3D features as the query vector and the local 3D features as the key vector and value vector into the first multi-head attention unit. The first multi-head attention unit processes the global 3D features and the local 3D features to obtain the initial 3D visual features.

[0155] In one embodiment, the method for determining a motion sequence model based on a android provided in this disclosure may include:

[0156] The calculation module is used to calculate the initial loss function value between the real action sequence indicated by the action sequence label and the reference action sequence;

[0157] The determination module is used to determine the loss function value based on the product of the initial loss function value and the keyframe weights;

[0158] The adjustment module is used to adjust the parameters of the module to be trained when the loss function value does not meet the training stopping condition, while keeping the backbone parameters of the action sequence model unchanged until the trained action sequence model meets the training stopping condition.

[0159] It is understood that the motion sequence model determination device based on android provided in the above embodiments, when implementing the corresponding motion sequence model determination method based on android, can allocate the above processing to different program modules as needed to complete all or part of the processing described above. Furthermore, the device and the corresponding method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0160] Based on the same inventive concept, this application also provides an action sequence determination device, specifically combined with... Figure 4 The action sequence determination device provided in the embodiments of this application will be described in detail.

[0161] Figure 4 This is a schematic diagram of the structure of an action sequence determination device provided in an embodiment of this application.

[0162] like Figure 4 As shown, the action sequence determination device 400 may include:

[0163] The acquisition module 410 is used to acquire actual image frame pairs from multiple different perspectives of the embodied robot in the actual scene, actual state frames associated with multiple actual image frame pairs, and actual task instructions.

[0164] Processing module 420 is used to input multiple actual image frame pairs, actual status frames, and actual task instructions into the processing module. Figure 1The method shown in the figure obtains a motion sequence model based on the embodied robot. In the motion sequence model obtained by the method, multiple actual image frame pairs, actual state frames and actual task instructions are processed by the motion sequence model to obtain the predicted motion sequence of the embodied robot.

[0165] It is understood that, when implementing the corresponding action sequence determination method, the action sequence determination device provided in the above embodiments can allocate the above processing to different program modules as needed to complete all or part of the processing described above. Furthermore, the device and the corresponding method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0166] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a method for determining a motion sequence model or a motion sequence determination method based on an embodied robot.

[0167] This application provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will execute the action sequence model determination method or action sequence determination method based on embodied robots provided in this application.

[0168] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0169] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0170] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0171] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0172] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure; as shown below. Figure 5 As shown, the electronic device 50 includes: a processor 501, and a memory 502 communicatively connected to the processor 501; the memory 502 stores instructions executable by the processor 501. The instructions are executed by the processor 501 to enable the processor 501 to perform:

[0173] Obtain training samples and corresponding action sequence labels. The action sequence labels represent the actual action sequences of the embodied robot corresponding to the training samples. The training samples include multiple image frame pairs of the embodied robot from multiple different perspectives of the target scene, state frames associated with multiple image frame pairs, and task instructions. Each image frame pair includes an RGB image frame and a depth image frame.

[0174] The action sequence model with parameter freezing is used to process multiple image frame pairs and the task instructions to obtain visual text feature information and key frame weights. The key frame weights are used to measure the importance of the state frames, and reference state feature information is determined based on the key frame weights and the state frames.

[0175] The visual text feature information and the reference state feature information are processed using the adaptation module to be trained to obtain the reference action sequence of the embodied robot;

[0176] During the training of the action sequence model, the parameters of the adaptation module to be trained are adjusted based on the loss function value determined by the action sequence label, the reference action sequence, and the keyframe weights, while keeping the backbone parameters of the action sequence model unchanged, until the trained action sequence model meets the training stopping condition.

[0177] Alternatively, the instruction is executed by processor 501 to enable processor 501 to perform the following:

[0178] The robot acquires actual image frame pairs from multiple different perspectives in a real-world scenario, actual state frames associated with the actual image frame pairs, and actual task instructions. Each actual image frame pair includes an actual RGB image frame and an actual depth image frame.

[0179] Multiple actual image frame pairs, actual state frames, and actual task instructions are input into an action sequence model obtained using the action sequence model determination method based on the embodied robot provided in the foregoing embodiments. The action sequence model is then used to process the multiple actual image frame pairs, actual state frames, and actual task instructions to obtain the predicted action sequence of the embodied robot.

[0180] The electronic devices provided in the above embodiments and the corresponding embodiments of the action sequence model determination method based on embodied robots or the action sequence determination method belong to the same concept. For details of their specific implementation process, please refer to the method embodiments, which will not be repeated here.

[0181] In practical applications, the electronic device 50 may further include at least one network interface 503. The various components of the electronic device 50 are coupled together via a bus system 504. It is understood that the bus system 504 is used to implement communication between these components. In addition to a data bus, the bus system 504 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 All buses are labeled as bus system 504. The number of processors 501 and the number of memories 502 can be at least one. The network interface 503 is used for wired or wireless communication between the electronic device 50 and other devices.

[0182] The memory 502 in this embodiment is used to store various types of data to support the operation of the electronic device 50.

[0183] The methods disclosed in the above embodiments of this disclosure can be applied to or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 501 or by instructions in the form of software. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 501 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this disclosure can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 502. Processor 501 reads the information in memory 502 and, in conjunction with its hardware, completes the steps of the aforementioned method for determining the action sequence model based on a hymenal robot or the steps of the action sequence determination method.

[0184] In some embodiments, the electronic device 50 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned methods.

[0185] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0186] In the above description, the term "some embodiments" refers to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0187] Unless otherwise defined, all technical and scientific terms used in this disclosure have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used in this disclosure is for the purpose of describing embodiments of this disclosure only and is not intended to be limiting of this disclosure.

[0188] It should be understood that in the various embodiments of this disclosure, the sequence number of each implementation process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0189] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0190] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for determining a motion sequence model based on an embodied robot, characterized in that, The method includes: Obtain training samples and corresponding action sequence labels. The action sequence labels represent the actual action sequences of the embodied robot corresponding to the training samples. The training samples include multiple image frame pairs of the embodied robot from multiple different perspectives of the target scene, state frames associated with multiple image frame pairs, and task instructions. Each image frame pair includes an RGB image frame and a depth image frame. The action sequence model with parameter freezing is used to process multiple image frame pairs and the task instructions to obtain visual text feature information and key frame weights. The key frame weights are used to measure the importance of the state frames, and reference state feature information is determined based on the key frame weights and the state frames. The visual text feature information and the reference state feature information are processed using the adaptation module to be trained to obtain the reference action sequence of the embodied robot; During the training of the action sequence model, the parameters of the adaptation module to be trained are adjusted based on the loss function value determined by the action sequence label, the reference action sequence, and the keyframe weights, while keeping the backbone parameters of the action sequence model unchanged, until the trained action sequence model meets the training stopping condition.

2. The method according to claim 1, characterized in that, The adaptation module to be trained includes a multimodal adapter and an action adapter; The process of using the adaptation module to be trained to process the visual text feature information and the reference state feature information to obtain the reference action sequence of the embodied robot includes: The visual text feature information is processed using the multimodal adapter to obtain processed visual text feature information, and reference visual text feature information is determined based on the visual text feature information and the processed visual text feature information. The motion prediction information of the android is processed using the motion adapter to obtain processed motion prediction information. Based on the motion prediction information and the processed motion prediction information, a reference motion sequence of the android is determined. The motion prediction information is determined based on the reference visual text feature information and the reference state feature information.

3. The method according to claim 1, characterized in that, The action sequence model includes a visual feature extraction module, a text feature extraction module, and a visual-text fusion module; The action sequence model using parameter freezing processes multiple image frame pairs and the task instructions to obtain visual text feature information and keyframe weights, including: The visual feature extraction module is used to process the RGB image frames and depth image frames in multiple image frame pairs to obtain three-dimensional visual features; The task instructions are processed using a text feature extraction module to obtain text features; The visual-text fusion module is used to process the three-dimensional visual features and the text features to obtain visual-text feature information and the keyframe weights.

4. The method according to claim 3, characterized in that, The visual feature extraction module includes a two-dimensional feature extraction submodule, a depth feature extraction submodule, a three-dimensional feature extraction submodule, and a visual feature fusion submodule; The step of processing RGB image frames and depth image frames from multiple image frame pairs using the visual feature extraction module to obtain three-dimensional visual features includes: The two-dimensional feature extraction submodule is used to extract features from the RGB image frames in multiple image frame pairs to obtain two-dimensional visual features. The depth feature extraction submodule is used to extract features from the depth image frames in multiple image frame pairs to obtain depth visual features. The three-dimensional feature extraction submodule is used to extract features from the RGBD information corresponding to multiple image frame pairs to obtain initial three-dimensional visual features. The RGBD information corresponding to each image frame pair is determined based on the RGB image frame and the depth image frame in the image frame pair. The visual feature fusion submodule is used to fuse the two-dimensional visual features, the depth visual features, and the initial three-dimensional visual features to obtain the three-dimensional visual features.

5. The method according to claim 4, characterized in that, The multiple image frame pairs include robotic arm image frame pairs and head image frame pairs; the three-dimensional feature extraction submodule includes an RGBD segmentation unit, a three-dimensional feature extraction unit, and a first multi-head attention unit; The step of using the three-dimensional feature extraction submodule to extract features from the RGBD information corresponding to multiple image frames to obtain initial three-dimensional visual features includes: The RGBD segmentation unit is used to segment the RGBD information corresponding to the image frames of the robotic arm to obtain the region information of the target object. The three-dimensional feature extraction unit is used to extract features from the region information of the target object to obtain local three-dimensional features; The three-dimensional feature extraction unit is used to perform feature extraction processing on the RGBD information corresponding to the head image frames to obtain global three-dimensional features; Using the global 3D features as the query vector and the local 3D features as the key vector and value vector, the global 3D features and the local 3D features are input into the first multi-head attention unit. The first multi-head attention unit processes the global 3D features and the local 3D features to obtain the initial 3D visual features.

6. The method according to claim 1, characterized in that, The step of adjusting the parameters of the adaptation module to be trained based on the loss function value determined by the action sequence label, the reference action sequence, and the keyframe weights, while keeping the backbone parameters of the action sequence model unchanged, until the trained action sequence model meets the training stopping condition, includes: Calculate the initial loss function value between the actual action sequence indicated by the action sequence label and the reference action sequence; The loss function value is determined based on the product of the initial loss function value and the keyframe weight; If the loss function value does not meet the training stopping condition, adjust the parameters of the adaptation module to be trained, and keep the backbone parameters of the action sequence model unchanged until the trained action sequence model meets the training stopping condition.

7. A method for determining an action sequence, characterized in that, The method includes: The robot acquires actual image frame pairs from multiple different perspectives in a real-world scenario, actual state frames associated with the actual image frame pairs, and actual task instructions. Each actual image frame pair includes an actual RGB image frame and an actual depth image frame. Multiple actual image frame pairs, actual state frames, and actual task instructions are input into an action sequence model obtained using the action sequence model determination method based on a hymenoid robot according to any one of claims 1 to 5. The action sequence model is then used to process the multiple actual image frame pairs, actual state frames, and actual task instructions to obtain the predicted action sequence of the hymenoid robot.

8. A device for determining a motion sequence model based on an embodied robot, characterized in that, The device includes: The acquisition module is used to acquire training samples and action sequence labels corresponding to the training samples. The action sequence labels represent the actual action sequences of the embodied robot corresponding to the training samples. The training samples include multiple image frame pairs of the embodied robot under multiple different perspectives of the target scene, status frames associated with multiple image frame pairs, and task instructions. Each image frame pair includes an RGB image frame and a depth image frame. The processing module is used to process multiple image frame pairs and the task instructions using a parameter-frozen action sequence model to obtain visual text feature information and key frame weights. The key frame weights are used to measure the importance of the state frames, and reference state feature information is determined based on the key frame weights and the state frames. The processing module is also used to process the visual text feature information and the reference state feature information using the adaptation module to be trained, so as to obtain the reference action sequence of the embodied robot. An adjustment module is used to adjust the parameters of the adaptation module to be trained based on the loss function value determined by the action sequence label, the reference action sequence, and the keyframe weights during the training process of the action sequence model, while keeping the backbone parameters of the action sequence model unchanged, until the trained action sequence model meets the training stopping condition.

9. An action sequence determination device, characterized in that, The device includes: The acquisition module is used to acquire actual image frame pairs of the embodied robot from multiple different perspectives in a real scene, actual state frames associated with multiple actual image frame pairs, and actual task instructions. Each actual image frame pair includes an actual RGB image frame and an actual depth image frame. The processing module is used to input multiple actual image frame pairs, actual state frames, and actual task instructions into an action sequence model obtained by the action sequence model determination method based on the embodied robot according to any one of claims 1 to 5, and to process the multiple actual image frame pairs, actual state frames, and actual task instructions through the action sequence model to obtain the predicted action sequence of the embodied robot.

10. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

Images