Embodied robot action prediction method and device based on multi-modal flow matching

Abstract

The application provides a kind of embodied robot action prediction method and device based on multimodal flow matching, it is related to the technical field of intelligent robot, including: obtaining instruction text and the image feature set collected by robot, and the depth image set corresponding to the image feature of each time position;Image feature set is handled to feature splicing and feature refining, and image sequence feature set is obtained, and based on image sequence feature set, feature fusion processing is carried out to depth image set, to determine target visual feature;Text feature in instruction text is fused with target visual feature, to determine text visual modal feature information, and based on text visual modal feature information, the action of robot manipulator is predicted, to determine motion pose prediction feature.The application can significantly improve the accuracy of embodied robot action prediction.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent robots, and in particular to a method and apparatus for predicting the motion of an embodied robot based on multimodal flow matching. Background Technology

[0002] Currently, related technologies suggest that when embodied robots perform motion prediction, they mainly rely on single-frame images as modal input and use a fixed strategy to predict robot actions. However, the above approach does not consider the information between video sequences, which can easily lead to information loss and thus result in poor motion prediction accuracy. In addition, the lack of depth information in the prediction process also prevents the model from accurately perceiving the environment in depth, thereby affecting the accuracy of the model's motion prediction. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method and apparatus for predicting the motion of an embodied robot based on multimodal flow matching, which can significantly improve the accuracy of motion prediction for embodied robots.

[0004] In a first aspect, embodiments of the present invention provide a method for predicting the motion of an embodied robot based on multimodal flow matching. The method includes: acquiring instruction text and a set of image features collected by the robot, as well as a set of depth images corresponding to the image features at each time position; performing feature stitching and feature refinement processing on the image feature set to obtain an image sequence feature set, and performing feature fusion processing on the depth image set based on the image sequence feature set to determine target visual features; determining text visual modal feature information by fusing text features in the instruction text with target visual features, and predicting the motion of the robot arm based on the text visual modal feature information to determine motion pose prediction features, wherein the motion pose prediction features are used to predict the motion pose of the robot arm at the next time step.

[0005] In one implementation, the steps of performing feature stitching and feature refinement on the image feature set to obtain the image sequence feature set include: obtaining an input feature set based on the image feature set, and using a three-view, three-frame spatiotemporal cross-attention model to perform feature stitching on the input feature set to determine environmental feedback features; and refining the environmental feedback features through a positive feedback module to determine the image sequence feature set.

[0006] In one implementation, the step of obtaining the input feature set based on the image feature set includes: adding the image feature sets collected by the cameras of the robot's head and left and right arms at three consecutive time points to the time-coded features of the corresponding time points to obtain the input feature set.

[0007] In one implementation, the step of performing feature fusion processing on a depth image set based on an image sequence feature set to determine the target visual features includes: performing semantic segmentation processing on the depth image set, the instruction text, and the image feature set respectively to obtain semantic segmentation results; and determining the category with the most votes as the current mask category by voting on the semantic segmentation results; extracting the visual variation features in the time channel range based on the current mask category to determine the category semantic features; and performing feature fusion processing on the semantic features and the image sequence feature set through a visual feature fusion module to determine the target visual features.

[0008] In one embodiment, the step of performing semantic segmentation processing on the depth image set, the instruction text, and the image feature set to obtain semantic segmentation results includes: rendering the depth image set and sending it to an unsupervised segmentation network to obtain a first semantic mask, and using an open vocabulary semantic segmentation network to perform mask recognition processing on the image features and instruction text corresponding to each depth image to obtain a second semantic mask, wherein the semantic segmentation results include: the first semantic mask and the second semantic mask.

[0009] In one implementation, the step of determining text visual modality feature information by fusing text features in the instruction text with target visual features includes: performing feature extraction and feature fusion processing on the instruction text using a unimodal text encoder and a multimodal text encoder respectively to obtain text features, and fusing the text features with target visual features using a cross-attention model and a self-attention model to determine the fused text visual modality feature information.

[0010] In one implementation, the step of predicting the motion of a robot arm based on text visual modal feature information and determining motion pose prediction features includes: encoding the robot arm motion features according to the text visual modal feature information to obtain a motion feature set, wherein the motion feature set includes: Gaussian noise encoding, robot arm motion frequency encoding, diffusion time step encoding, robot arm motion feature encoding, and robot arm current pose encoding; determining the conditional encoding of the flow matching diffusion model based on the robot arm current pose encoding, and combining it with the remaining motion features to obtain input encoding features, so that the flow matching diffusion model determines the motion pose prediction features based on the conditional encoding and the input encoding features.

[0011] Secondly, embodiments of the present invention also provide a motion prediction device for an embodied robot based on multimodal flow matching. The device includes: an information acquisition module, which acquires instruction text and a set of image features collected by the robot, as well as a set of depth images corresponding to the image features at each time position; a feature extraction module, which performs feature splicing and feature refinement processing on the image feature set to obtain an image sequence feature set, and performs feature fusion processing on the depth image set based on the image sequence feature set to determine the target visual features; and a pose prediction module, which determines text visual modal feature information by fusing text features in the instruction text with target visual features, and predicts the motion of the robot arm based on the text visual modal feature information to determine motion pose prediction features, wherein the motion pose prediction features are used to predict the motion pose of the robot arm at the next time moment.

[0012] Thirdly, embodiments of the present invention also provide a server, including a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement any of the methods provided in the first aspect.

[0013] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement any of the methods provided in the first aspect.

[0014] The embodiments of the present invention bring the following beneficial effects: This invention provides a method and apparatus for predicting the actions of an embodied robot based on multimodal flow matching. The method acquires instruction text, a set of image features collected by the robot, and a set of depth images corresponding to the image features at each time point. It then performs feature stitching and refinement on the image feature sets to obtain an image sequence feature set. Based on this image sequence feature set, it performs feature fusion on the depth image set to determine the target visual features. Finally, it fuses the text features in the instruction text with the target visual features to determine text visual modal feature information. Based on this text visual modal feature information, it predicts the robot arm's actions and determines the motion pose prediction features. This invention improves the model's ability to perceive the environment by incorporating image depth information and image sequence information, while also providing environmental feedback information to enhance the accuracy of the model's robot action prediction. Furthermore, it improves the accuracy of the model's action prediction under different paths through a flow matching mechanism network.

[0015] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0016] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a method for predicting the motion of an embodied robot based on multimodal flow matching, provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a model training and inference method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a body robot motion prediction device based on multimodal flow matching provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Currently, related technologies propose that embodied robots mainly rely on single-frame images as modal inputs for motion prediction, employing fixed strategies to predict robot actions. However, these schemes, by neglecting information between video sequences, are prone to information loss, resulting in poor motion prediction accuracy. Furthermore, the lack of depth information during prediction also prevents the model from accurately perceiving the operating environment, thus affecting the accuracy of robot motion prediction. In other words, the main problem with existing robot motion prediction schemes lies in the fact that the model only considers single-frame visual information, and some visual modal inputs fail to consider depth information. Based on this, the embodied robot motion prediction method and apparatus based on multimodal flow matching provided by this invention can improve the model's ability to perceive the environment by incorporating image depth information and image sequence information, while providing environmental feedback information to enhance the accuracy of robot motion prediction. In addition, the flow matching mechanism network improves the accuracy of the model in adapting to different paths for motion prediction.

[0021] See Figure 1 The diagram shows a flowchart of a method for predicting the motion of an embodied robot based on multimodal flow matching. The method mainly includes the following steps S102 to S106: Step S102: Obtain the instruction text and the set of image features acquired by the robot, as well as the set of depth images corresponding to the image features at each time point. The set of image features includes: RGB images obtained from the robot's left arm camera. The head-mounted camera obtains RGB images. The right arm camera obtains RGB images. The depth image includes the corresponding left arm depth image. Head depth image Depth image of the right arm In one implementation, it is necessary to simultaneously acquire RGB images and corresponding depth images at different times from each camera at times t, t+1, and t+2 on the time series axis, where t, t+1, and t+2 are spaced 30 frames apart.

[0022] Step S104 involves performing feature stitching and feature refinement on the image feature set to obtain an image sequence feature set. Based on the image sequence feature set, feature fusion processing is performed on the depth image set to determine the target visual features. In one implementation, an input feature set can be obtained based on the image feature set. A three-view, three-frame spatiotemporal cross-attention model is used to perform feature stitching on the input feature set to determine environmental feedback features. Then, the environmental feedback features are refined through a positive feedback module to determine the image sequence feature set. Specifically, the image feature sets collected by the cameras on the robot's head and left and right arms at three consecutive moments (each moment is 30 frames) can be added to the temporal coding features at the corresponding moments to obtain the input feature set.

[0023] In practical applications, the RGB images acquired at different times and locations can be input into the BLIP2 image feature extractor ImageEncoder. The parameters in this feature extractor are frozen and do not participate in model training, thus obtaining the left arm image features at time t. Left arm image features at time t+1 Left arm image features at time t+2 Head image features at time t Head image features at time t+1 Head image features at time t+2 Right arm image features at time t Right arm image features at time t+1 Right arm image features at time t+2 The RGB features at different locations are input as a unit (i.e., the input feature set) into the VSE Module (i.e., the spatiotemporal cross-attention model with three views and three frames). This unit contains features at time t, time t+1, and time t+2.

[0024] First, the characteristics at time t are... The feature is obtained by adding the time-encoded feature at time t. Image features at time t+1 The feature is obtained by adding the time-encoded feature at time t+1. Image features at time t+2 The feature is obtained by adding the time-coded feature at time t+2. and will , and features Concatenate to merge, then tile to obtain features. At the same time, set the learnable parameter `learned latent quries`. With features Concatenate (used to join two or more tensors together along a certain dimension to generate a larger tensor) to obtain features. This parameter serves as the key, and the value is... The features are obtained by inputting the question into the Cross-Attention module. (That is, environmental feedback features), which integrate more visual information at different times, helps the model obtain environmental feedback information when performing actions, and helps the model make the next prediction information.

[0025] Furthermore, to enable better learning and fitting of features for fusion with subsequent features, these features need to be input into the FFW positive feedback module (a stage that refines the "temporal-multi-view" visual features fused by cross-attention, while using residual structures to prevent information loss, providing clean, compact, and highly expressive visual features for higher-order fusion / decision networks). This module consists of multiple MLPs, ReLU activation functions, and Norm normalization modules in sequence, and incorporates a residual mechanism to prevent the model from forgetting specific operations. This module first incorporates environmental feedback features. With features Adding them together yields the features. Simultaneously, the features are input into the first FFW module. , and features Adding together yields the features This feature is then input into the second FFW module to obtain the feature. Since the feature sets of image sequences of the robot's left arm, head, and right arm are input into this module respectively, these three features are denoted as follows: , , .

[0026] Step S106: By fusing text features in the instruction text with target visual features, text visual modal feature information is determined. Based on the text visual modal feature information, the robot arm's movements are predicted to determine motion pose prediction features. The motion pose prediction features are used to predict the robot arm's motion pose at the next moment. In one embodiment, the robot arm's motion features can be encoded based on the text visual modal feature information to obtain a motion feature set. The motion feature set includes: Gaussian noise encoding, robot arm motion frequency encoding, diffusion time step encoding, robot arm action feature encoding, and robot arm current pose encoding. Then, the conditional encoding of the flow matching diffusion model is determined based on the robot arm's current pose encoding, and the remaining motion features are combined to obtain input encoding features. The flow matching diffusion model determines the motion pose prediction features based on the conditional encoding and input encoding features.

[0027] In practical applications, the fused text visual modality feature information can be input into the ACUFM Module, which encodes Gaussian noise to obtain features. The characteristics are obtained by encoding the motion frequency of the robotic arm. Feature encoding is performed on the diffusion time step to obtain features. The features are obtained by encoding the motion features of the robotic arm. Finally, the current pose of the robotic arm is encoded to obtain features. .

[0028] In one implementation, the step of combining motion features to obtain input encoded features specifically includes: firstly, combining motion frequency features... With features The features extracted from the MLP module are concatenated and merged to obtain the final feature. ,use , to obtain features At the same time Obtain features After that With features Adding them together yields the features. (That is, input encoded features), then the current pose of the robotic arm. Obtained through MLP encoding , as a conditional encoding for the flow-matching diffusion model.

[0029] Furthermore, the results obtained above... After MLP module and features Concatenate to obtain features Then, the features are obtained through another MLP module. , the features obtained previously After the MLP module is used for features Concatenating and then processing the FMUNetBlock module to obtain features The FMUNetBlock module first processes the input features... The inputs are fed into Reshape Module1 and Reshape Module2 respectively, and these two modules perform reshape operations to obtain the features. and features , will feature The input is fed into two downsampling modules, each consisting of multiple convolutions, pooling, normalization, and activation functions to obtain features. Then, this feature is compared with the previously obtained features. Features are obtained by performing CrossAttention operation. After a subsequent upsampling, the features were obtained. Furthermore, this feature is consecutively input into three upsampling modules, the UnSampBlock module, to obtain the feature. As k, v and the previously obtained features The features are input together with q into the Cross-Attention module to obtain the features. The features are then obtained by inputting them into the self-attention mechanism. The aforementioned UnsampBlock consists of multiple deconvolutions, normalizations, and activation functions, which are then input into the Head layer to obtain the features. (That is, motion pose prediction feature), this head layer consists of multiple MLPs, activation functions, and normalization. This feature is used to predict the pose that the robotic arm needs to move next, so it is used to calculate the loss function. Here we choose MSE as the loss function.

[0030] The above-described embodied robot action prediction method based on multimodal flow matching provided in this invention can improve the overall generalization performance of the model, thereby enabling the model to still have good execution performance even in scenarios with a small number of training samples. In addition, by using unsupervised segmentation information, the model's ability to understand the spatial semantics of the object to be manipulated is improved, thereby improving the accuracy of the model's prediction execution. Furthermore, by using depth image and video sequence information, the model's ability to perceive the environment and receive environmental feedback information is improved, thereby enhancing the accuracy of the model's prediction execution. Finally, through the flow matching mechanism, the accuracy of the model in predicting actions under different paths is improved.

[0031] This invention also provides an implementation method for determining the visual features of a target, as detailed in (1) to (3) below: (1) Semantic segmentation processing is performed on the depth image set, the instruction text and the image feature set respectively to obtain the semantic segmentation result. The category with the most votes is determined as the current mask category by voting on the semantic segmentation result. In one embodiment, the depth image set can be rendered and sent to an unsupervised segmentation network to obtain the first semantic mask. The open vocabulary semantic segmentation network is used to perform mask recognition processing on the image features and instruction text corresponding to each depth image to obtain the second semantic mask. The semantic segmentation result includes the first semantic mask and the second semantic mask.

[0032] In practical applications, for depth image information, it is first converted into Colormaps to obtain the rendered depth map, denoted as . The result is then fed into the SAM2 unsupervised segmentation network to obtain the first semantic mask. The network weights are completely frozen and do not participate in model training. At the same time, the RGB image at the corresponding time position and the instruction text are input into the OVSeg (Open-Vocabulary Semantic) network to obtain the second semantic mask. OVSeg performs zero-shot semantic segmentation on RGB images. It identifies the category of objects based on the object categories contained in the input text information. Then, the category of each SAM mask is determined by voting based on the semantic segmentation results of the points within the current mask. The category with the most points is selected as the category of the current mask (a matrix or image of the same size as the original image, used to indicate or identify pixels or regions in the original image). This feature is denoted as... (That is, visual information features), which not only include the semantic category information of vision, but also the spatial position information of different objects in vision, providing the model with more accurate spatial prior information for precise grasping.

[0033] Furthermore, the features obtained from the robotic arm at different positions are concatenated and then input into the 3DCNN convolutional module. This module can extract visual information at different times, and its operators can also obtain the visual variation features over time. These features are denoted as... Category semantic features, and finally the left arm image sequence features, head image sequence features, right arm image sequence features, and category semantic features are combined. They are input together into the Vision FeatureFusion Module (i.e., the visual feature fusion module).

[0034] (2) Based on the current mask category, the visual variation features in the time channel range are extracted to determine the category semantic features. Then, through the visual feature fusion module, the semantic features and the image sequence feature set are fused to determine the target visual features. In one implementation, for the visual feature fusion module, the image sequence features are first input into their respective CBAM channel spaces, and the attention mechanism module obtains the features respectively. , and features The data are then fed into the ConvBlock convolutional module, which contains a combination of multiple convolution operations, activation functions, normalization operations, and pooling operations to ultimately obtain the features. , and features Then, a residual mechanism is added to prevent information forgetting, and the obtained features are respectively... , , With features ,feature and features Performing the Concate operation yields the features. , , These three features represent the more semantically rich visual information from the left arm's perspective, the head's perspective, and the right arm's perspective, respectively. Since the head's perspective provides more comprehensive visual information, while the left and right arm perspectives provide more detailed visual information, a cross-attention operation is performed between the head's visual information and the detailed visual information from the left and right arms. The head's visual information is represented by q, and the left and right arm visual information by k and v, respectively. The final feature obtained is... and features and associate this feature with The three are concatenated to obtain the characteristic. Then we will use the previously obtained category semantic features. With features Features are obtained by performing a cross-attention operation. This operation better integrates spatial visual information with semantic spatial information, resulting in richer visual information, including category semantic features. We are k and v, features As q, the features obtained later Features are obtained by inputting into the Self-Attention mechanism. (i.e., target visual features).

[0035] (3) The text is processed by feature extraction and feature fusion of the instruction text by a unimodal text encoder and a multimodal text encoder respectively to obtain text features. The text features are then fused with the target visual features by a cross-attention model and a self-attention model to determine the fused text visual modality feature information. In one implementation, for text information, the text instruction is first input into the BLIP2 unimodal text encoder and the multimodal text encoder respectively to obtain features. and features Then, the features of the two are concatenated to obtain the final features. (That is, text features), and then we need to consider the fusion of text and visual modalities. Through the IT Fusion Module, we first input the features into the MLP module to obtain the features. This feature Consider it as q, compared with the previously obtained target visual features. Both k and v are considered as inputs into a cross-attention mechanism for the initial fusion of textual and visual modalities to obtain features. Then, the feature is obtained by performing a Flatten exhibit operation on the feature. The features are then input again into the GAP global average pooling operation. .

[0036] Furthermore, this feature is input into the MLP module to obtain features. The weighted features are obtained by concatenating the Simgoid activation function. Because the fusion of text and visual modalities requires consideration of the respective weights' influence on the final feature, this feature effectively balances the importance of both modalities. Therefore, the previously obtained feature will be used in the subsequent analysis. With weight features Multiplication yields the features and features obtained previously Adding them together yields the features. The features are then input into the Self-Attention module to obtain the features. And the features obtained by the MLP module This feature integrates textual and visual information features.

[0037] See Figure 2 The diagram illustrates a model training and inference method. This embodiment of the invention also provides an implementation method for model training and inference. During model training, the dataset is divided into a training set and a validation set. The training set is used during training, while the validation set is used to observe the convergence of model training. The weights with the lowest decrease in the loss function of the validation set are selected as the final weights of the model.

[0038] In practical applications, MSE can be selected as the model training loss function. During inference, the current robotic arm motion is set as Gaussian random noise as the model input. The model's prediction is the Gaussian random noise plus the predicted value multiplied by the position code of the time diffusion step t as the final prediction result. In addition, random masking and image augmentation are performed on the image during training, including color jitter and arbitrary cropping, which helps the model adapt to different lighting conditions. At the same time, the model does not rely too much on information from a single camera, but can use the common information from three cameras for behavior prediction. Finally, in the prediction execution phase, the robotic arm will traverse and execute according to the motion sequence predicted by the model, and then continuously perform model inference until the model completes the overall action.

[0039] In summary, this invention can improve the accuracy and generalization of the model in new scenarios through the embodied algorithm diffusion model AFM Network; provide the algorithm with image detail features and visual depth information through the 3DSS spatial semantic prior feature extraction module, thereby enabling the robot arm to move precisely and improving motion execution accuracy; extract visual sequence information more fully through the VSE Module, and improve the model's ability to perceive the environment by adding image depth information and image sequence information, while providing environmental feedback information to improve the model's prediction success rate; and improve the accuracy of the model in predicting actions under different paths through the FMUNet flow matching mechanism network.

[0040] Regarding the android motion prediction method based on multimodal flow matching provided in the foregoing embodiments, this invention provides an android motion prediction device based on multimodal flow matching. See [link to relevant documentation]. Figure 3 The diagram shows a structural schematic of an embodied robot motion prediction device based on multimodal flow matching. The device includes the following parts: The information acquisition module 302 acquires the instruction text and the set of image features collected by the robot, as well as the set of depth images corresponding to the image features at each time position; The feature extraction module 304 performs feature splicing and feature refinement on the image feature set to obtain the image sequence feature set, and performs feature fusion on the depth image set based on the image sequence feature set to determine the target visual features. The pose prediction module 306 determines the text visual modal feature information by fusing the text features in the instruction text with the target visual features, and predicts the movement of the robot arm based on the text visual modal feature information to determine the motion pose prediction features. The motion pose prediction features are used to predict the motion pose of the robot arm at the next moment.

[0041] The above-described embodied robot motion prediction device based on multimodal flow matching provided in this application embodiment can significantly improve the accuracy of embodied robot motion prediction.

[0042] In one embodiment, when performing feature stitching and feature refinement processing on the image feature set to obtain the image sequence feature set, the feature extraction module 304 is further configured to: obtain an input feature set based on the image feature set, and use a three-view, three-frame spatiotemporal cross-attention model to perform feature stitching processing on the input feature set to determine environmental feedback features; and refine the environmental feedback features through a positive feedback module to determine the image sequence feature set.

[0043] In one embodiment, when performing the step of obtaining the input feature set based on the image feature set, the feature extraction module 304 is further configured to: add the image feature sets collected by the cameras of the robot's head and left and right arms at three consecutive time moments to the time-coded features at the corresponding time moments to obtain the input feature set.

[0044] In one embodiment, when performing feature fusion processing on a depth image set based on an image sequence feature set to determine the target visual features, the feature extraction module 304 is further configured to: perform semantic segmentation processing on the depth image set, the instruction text, and the image feature set respectively to obtain semantic segmentation results; and determine the category with the most votes as the current mask category by voting on the semantic segmentation results; extract the visual variation features in the time channel range based on the current mask category to determine the category semantic features; and perform feature fusion processing on the semantic features and the image sequence feature set through the visual feature fusion module to determine the target visual features.

[0045] In one embodiment, when performing semantic segmentation processing on the depth image set, the instruction text, and the image feature set to obtain semantic segmentation results, the feature extraction module 304 is further configured to: render the depth image set and send it to an unsupervised segmentation network to obtain a first semantic mask, and use an open vocabulary semantic segmentation network to perform mask recognition processing on the image features and instruction text corresponding to each depth image to obtain a second semantic mask, wherein the semantic segmentation results include: the first semantic mask and the second semantic mask.

[0046] In one embodiment, when performing the step of determining text visual modality feature information by fusing text features in the instruction text with target visual features, the pose prediction module 306 is further configured to: perform feature extraction and feature fusion processing on the instruction text by using a single-modal text encoder and a multimodal text encoder respectively to obtain text features, and fuse the text features with the target visual features by using a cross-attention model and a self-attention model to determine the fused text visual modality feature information.

[0047] In one embodiment, when performing the step of predicting the robot arm's movements based on text visual modal feature information and determining motion pose prediction features, the pose prediction module 306 is further configured to: encode the robot arm's motion features based on the text visual modal feature information to obtain a motion feature set, wherein the motion feature set includes: Gaussian noise encoding, robot arm motion frequency encoding, diffusion time step encoding, robot arm action feature encoding, and robot arm current pose encoding; determine the conditional encoding of the flow matching diffusion model based on the robot arm current pose encoding, and combine it with the remaining motion features to obtain input encoding features, so that the flow matching diffusion model determines the motion pose prediction features based on the conditional encoding and the input encoding features.

[0048] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0049] This invention provides a server, specifically, the server includes a processor and a storage device; the storage device stores a computer program, which, when run by the processor, executes the method described in any of the above embodiments.

[0050] Figure 4 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. The server 100 includes: a processor 40, a memory 41, a bus 42 and a communication interface 43. The processor 40, the communication interface 43 and the memory 41 are connected through the bus 42. The processor 40 is used to execute executable modules, such as computer programs, stored in the memory 41.

[0051] The memory 41 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 43 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0052] Bus 42 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0053] The memory 41 is used to store programs. After receiving an execution instruction, the processor 40 executes the program. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 40 or implemented by the processor 40.

[0054] Processor 40 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 40 or by instructions in software form. Processor 40 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 41. The processor 40 reads the information in memory 41 and, in conjunction with its hardware, completes the steps of the above method.

[0055] The computer program product of the readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, please refer to the foregoing method embodiments, which will not be repeated here.

[0056] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0057] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for predicting the motion of an embodied robot based on multimodal flow matching, characterized in that, The method includes: Acquire the instruction text and the set of image features collected by the robot, as well as the set of depth images corresponding to the image features at each time point; The image feature set is subjected to feature stitching and feature refinement to obtain an image sequence feature set. Based on the image sequence feature set, the depth image set is subjected to feature fusion to determine the target visual features. By fusing the text features in the instruction text with the target visual features, text visual modal feature information is determined, and based on the text visual modal feature information, the movement of the robot arm is predicted to determine the motion pose prediction feature, wherein the motion pose prediction feature is used to predict the motion pose of the robot arm at the next moment. The step of performing feature fusion processing on the depth image set based on the image sequence feature set to determine the target visual features includes: performing semantic segmentation processing on the depth image set, the instruction text, and the image feature set respectively to obtain semantic segmentation results; and determining the category with the most votes as the current mask category by voting on the semantic segmentation results; extracting the variation features of visual features in the time channel range based on the current mask category to determine semantic features; and performing feature fusion processing on the semantic features and the image sequence feature set through a visual feature fusion module to determine the target visual features. The step of performing semantic segmentation processing on the depth image set, the instruction text, and the image feature set to obtain semantic segmentation results includes: rendering the depth image set and sending it to an unsupervised segmentation network to obtain a first semantic mask; and using an open vocabulary semantic segmentation network to perform mask recognition processing on the image features and instruction text corresponding to each depth image to obtain a second semantic mask. The semantic segmentation results include the first semantic mask and the second semantic mask. The step of predicting the robot arm's movements and determining motion pose prediction features based on the text visual modal feature information includes: encoding the robot arm's motion features according to the text visual modal feature information to obtain a motion feature set, wherein the motion feature set includes: Gaussian noise encoding, robot arm motion frequency encoding, diffusion time step encoding, robot arm action feature encoding, and robot arm current pose encoding; determining the conditional encoding of the flow matching diffusion model based on the robot arm current pose encoding, and combining it with the remaining motion features to obtain input encoding features, so that the flow matching diffusion model determines the motion pose prediction features based on the conditional encoding and the input encoding features.

2. The method for predicting the motion of an embodied robot based on multimodal flow matching according to claim 1, characterized in that, The steps of performing feature concatenation and feature refinement on the image feature set to obtain the image sequence feature set include: Based on the image feature set, an input feature set is obtained, and a three-view, three-frame spatiotemporal cross-attention model is used to perform feature stitching on the input feature set to determine the environmental feedback features. The environmental feedback features are refined using a positive feedback module to determine the image sequence feature set.

3. The method for predicting the motion of an embodied robot based on multimodal flow matching according to claim 2, characterized in that, The step of obtaining the input feature set based on the image feature set includes: The input feature set is obtained by adding the image feature set collected by the cameras on the robot's head and left and right arms at three consecutive time points to the time-coded features at the corresponding time points.

4. The method for predicting the motion of an embodied robot based on multimodal flow matching according to claim 1, characterized in that, The step of determining text visual modality feature information by fusing text features in the instruction text with target visual features includes: The instruction text is processed by feature extraction and feature fusion using a unimodal text encoder and a multimodal text encoder respectively to obtain text features. The text features are then fused with the target visual features using a cross-attention model and a self-attention model to determine the fused text visual modality feature information.

5. A motion prediction device for embodied robots based on multimodal flow matching, characterized in that, The device includes: The information acquisition module acquires the instruction text and the set of image features collected by the robot, as well as the set of depth images corresponding to the image features at each time point and location; The feature extraction module performs feature concatenation and feature refinement on the image feature set to obtain an image sequence feature set, and performs feature fusion on the depth image set based on the image sequence feature set to determine the target visual features. The pose prediction module determines text visual modal feature information by fusing text features in the instruction text with target visual features, and predicts the movement of the robot arm based on the text visual modal feature information to determine motion pose prediction features, wherein the motion pose prediction features are used to predict the motion pose of the robot arm at the next moment. The step of performing feature fusion processing on the depth image set based on the image sequence feature set to determine the target visual features includes: performing semantic segmentation processing on the depth image set, the instruction text, and the image feature set respectively to obtain semantic segmentation results; and determining the category with the most votes as the current mask category by voting on the semantic segmentation results; extracting the variation features of visual features in the time channel range based on the current mask category to determine semantic features; and performing feature fusion processing on the semantic features and the image sequence feature set through a visual feature fusion module to determine the target visual features. The step of performing semantic segmentation processing on the depth image set, the instruction text, and the image feature set to obtain semantic segmentation results includes: rendering the depth image set and sending it to an unsupervised segmentation network to obtain a first semantic mask; and using an open vocabulary semantic segmentation network to perform mask recognition processing on the image features and instruction text corresponding to each depth image to obtain a second semantic mask. The semantic segmentation results include the first semantic mask and the second semantic mask. The step of predicting the robot arm's movements and determining motion pose prediction features based on the text visual modal feature information includes: encoding the robot arm's motion features according to the text visual modal feature information to obtain a motion feature set, wherein the motion feature set includes: Gaussian noise encoding, robot arm motion frequency encoding, diffusion time step encoding, robot arm action feature encoding, and robot arm current pose encoding; determining the conditional encoding of the flow matching diffusion model based on the robot arm current pose encoding, and combining it with the remaining motion features to obtain input encoding features, so that the flow matching diffusion model determines the motion pose prediction features based on the conditional encoding and the input encoding features.

6. A server, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method described in any one of claims 1 to 4.

Images