Motion determination methods and devices, robot systems, storage media, electronic devices

Abstract

This application provides an action determination method and apparatus, a robot system, a storage medium, and an electronic device, relating to the field of robotics. The method includes: acquiring task instructions and multi-view image data, wherein the multi-view image data includes image data collected by the robot from M views, where M is an integer greater than or equal to 2; determining a feature set based on the multi-view image data, wherein the feature set includes a semantic feature subset, a geometric feature subset, and a three-dimensional spatial feature subset corresponding to the M views; and determining the robot's execution action based on the feature set and the task instructions. This solves the problem of inaccurate robot execution actions in related technologies, enabling action generation based on accurate three-dimensional spatial understanding and improving the accuracy of execution actions.

Description

Technical Field

[0001] This application relates to the field of robotics, and more specifically, to a motion determination method and apparatus, a robot system, a storage medium, and an electronic device. Background Technology

[0002] In related technologies, mainstream robot vision-language-action (VLA) models generally rely on single-viewpoint or planar 2D visual input. They directly parse task instructions from red-green-blue (RGB) images and generate end-effector actions through end-to-end neural networks. While these methods demonstrate some instruction-following capabilities in simple, static, and low-occlusion scenarios, they essentially forcibly project the 3D physical world onto a 2D image space for modeling, lacking an explicit understanding of the real spatial structure, leading to inaccurate robot actions. Specifically, because action prediction is based solely on planar image features, the model struggles to accurately determine the 3D spatial position, depth relationship, and occlusion state of target objects. For example, when grasping a partially occluded cup, it may mistakenly grasp objects behind it due to the inability to distinguish the depth levels of the foreground and background. In scenarios with multiple targets juxtaposed, planar features can easily lead to target confusion, causing the robot to lose targets or repeat actions during continuous operations.

[0003] No effective solution has yet been proposed to address the problem of inaccurate robot actions in related technologies. Summary of the Invention

[0004] This application provides a motion determination method and apparatus, a robot system, a storage medium, and an electronic device to at least solve the problem of inaccurate robot motion execution in related technologies.

[0005] According to one embodiment of this application, an action determination method is provided, comprising: acquiring task instructions and multi-view image data, wherein the multi-view image data includes image data collected by a robot from M views, where M is an integer greater than or equal to 2; determining a feature set based on the multi-view image data, wherein the feature set includes a semantic feature subset, a geometric feature subset, and a three-dimensional spatial feature subset corresponding to the M views; and determining the robot's execution action based on the feature set and the task instructions.

[0006] According to another embodiment of this application, an action determination device is provided, comprising: an acquisition module for acquiring task instructions and multi-view image data, wherein the multi-view image data includes image data collected by a robot from M views, where M is an integer greater than or equal to 2; a first determination module for determining a feature set based on the multi-view image data, wherein the feature set includes a semantic feature subset, a geometric feature subset, and a three-dimensional spatial feature subset corresponding to the M views; and a second determination module for determining the robot's execution action based on the feature set and the task instructions.

[0007] According to yet another embodiment of this application, a robot system is also provided, including: a robotic arm, a control terminal, and a plurality of image acquisition devices, wherein the control terminal is configured to perform the steps in any of the above method embodiments.

[0008] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0009] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0010] According to yet another embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0011] Through the above embodiments of this application, task instructions and multi-view image data collected by the robot from multiple perspectives are obtained. By extracting the three-dimensional spatial features, semantic features, and geometric features of each perspective from the multi-view image data, since the three-dimensional spatial features can characterize the position of the target object in the real three-dimensional space, and combined with the semantic features and geometric features, cross-view and cross-modal spatial consistency perception is achieved, thereby enabling action generation based on accurate three-dimensional spatial understanding. This solves the problem of inaccurate robot action execution in related technologies and improves the accuracy of action execution. Attached Figure Description

[0012] Figure 1 This is a hardware structure block diagram of the robot system operated in the embodiments of the method of this application;

[0013] Figure 2 This is a flowchart of an action determination method according to an embodiment of this application;

[0014] Figure 3 This is a schematic diagram of a cross-view aligner and a cross-target fusion unit according to embodiments of this application;

[0015] Figure 4 This is an interactive schematic diagram of a cross-view aligner and a cross-scene thinker in an action determination method according to an optional embodiment of this application;

[0016] Figure 5 This is an interactive schematic diagram of a cross-target fusion device and a cross-scene thinker in an action determination method according to an optional embodiment of this application;

[0017] Figure 6 This is a schematic diagram of a cross-scenario thinker according to an embodiment of this application;

[0018] Figure 7 This is a schematic diagram of the structure of an action determination device according to an embodiment of this application. Detailed Implementation

[0019] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0020] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0021] The methods and embodiments provided in this application can be executed in a robot system. Taking running on a robot system as an example, Figure 1 This is a hardware structure block diagram of the robot system used in the embodiments of the method of this application. For example... Figure 1 As shown, the robot system may include two robotic arms 102, a control terminal 104 for controlling the robotic arms to perform actions, and three image acquisition devices 106 for acquiring images, wherein the three image acquisition devices 106 correspond to the main view, left view, and right view, respectively. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the robot system described above. For example, the robot system may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0022] The robotic arm 102, acting as an actuator, receives motion commands from the control terminal 104 and precisely drives the end effector (such as a gripper) to complete physical operations such as grasping, moving, and assembling. The joint encoders and torque sensors of the robotic arm 102 provide feedback on current posture and contact force information, forming a closed-loop control system to ensure the accuracy and safety of motion execution. The robotic arm 102 and the control terminal 104 are connected via a low-latency bus, achieving millisecond-level command response.

[0023] The control terminal 104 includes a high-performance processor and a high-speed memory. It receives multi-view RGB images acquired from multiple image acquisition devices 106 and externally input task commands, ultimately generating a continuous sequence of actions including translation, rotation, and gripper control. This sequence is then output to the robotic arm 102 via a real-time communication interface, achieving closed-loop decision-making from perception to action. Optionally, the control terminal 104 includes a central processing unit and a graphics processing unit, supporting real-time inference of deep learning models.

[0024] Image acquisition device 106 includes, but is not limited to, an RGB camera fixedly mounted on the robot body, responsible for synchronously acquiring environmental images during robot operation to form a multi-view visual input stream. Image acquisition device 106 transmits image data to control terminal 104 in real time via wired or wireless means, providing raw data support for semantic recognition, 3D reconstruction and cross-view alignment.

[0025] This embodiment provides a motion determination method for the aforementioned robot system. Figure 2 This is a flowchart of an action determination method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps S202 to S206:

[0026] Step S202: Obtain task instructions and multi-view image data, wherein the multi-view image data includes image data collected by the robot from M perspectives, where M is an integer greater than or equal to 2;

[0027] For example, the task instruction is a description of the user's intention to operate in natural language, such as "pick up the red water glass on the table and put it in the cabinet on the right", "fold this blue shirt and put it on the hanger", or "take the toolbox from the third shelf and hand it to me", etc. At the same time, multi-view image data I={0,L,R} is synchronously collected by multiple fixed or adjustable RGB cameras deployed on the robot body. For example, the main view (0) is installed in front of the robot's head or chest and is used to align the central field of view of the operation area; the left view (L) is located on the left side of the robot and is used to capture the side information of the occluded target; the right view (R) is located on the right side of the robot and provides supplementary observation symmetrical to the left view. The three together constitute a stereo vision system covering the operation area.

[0028] Step S204: Determine a feature set based on the multi-view image data, wherein the feature set includes a semantic feature subset, a geometric feature subset, and a three-dimensional spatial feature subset corresponding to the M views;

[0029] Optionally, the RGB image of the m-th viewpoint is processed according to an image semantic feature model (e.g., SigLIP). Encode to obtain semantic features : ; This represents the text encoder of SigLIP;

[0030] Optionally, the RGB image at the m-th viewpoint is processed according to an image geometric feature model (e.g., DINOv2). Encode to obtain geometric features : ; This refers to the visual encoder of DINOv2;

[0031] Optionally, the RGB image of the m-th viewpoint is processed by a visual encoder based on an image depth feature model (e.g., VGGT). Encode to obtain three-dimensional spatial features : , This refers to the visual encoder of VGGT.

[0032] Where DPT() represents dense prediction head, Represents a depth map. Represents a point graph. This indicates point tracking or cross-view correspondence features.

[0033] It should be noted that the semantic feature subset includes the semantic features of each of the M viewpoints, the geometric feature subset includes the geometric features of each of the M viewpoints, and the 3D space feature subset includes the 3D space features of each of the M viewpoints.

[0034] Step S206: Determine the robot's execution action based on the feature set and the task instruction.

[0035] Through the above steps, task instructions and multi-view image data collected by the robot from multiple perspectives are obtained. By extracting the three-dimensional spatial features, semantic features, and geometric features of each perspective from the multi-view image data, since the three-dimensional spatial features can characterize the position of the target object in the real three-dimensional space, and combined with the semantic features and geometric features, cross-view and cross-modal spatial consistency perception is achieved. In this way, the action generation is based on accurate three-dimensional spatial understanding, which solves the problem of inaccurate robot action execution in related technologies and improves the accuracy of action execution.

[0036] Optionally, the above-mentioned determination of the feature set based on the multi-view image data and determination of the robot's execution action based on the feature set and the task instruction includes: determining the feature set based on the multi-view image data using a target action prediction model, and determining the robot's execution action based on the feature set and the task instruction.

[0037] In other words, this application can determine the robot's execution actions through a target action prediction model. The target action prediction model includes: an image semantic feature model, an image geometric feature model, an image depth feature model, a cross-view alignment unit, a cross-target fusion unit, and a cross-scene thinker. The functions of the image semantic feature model, the image geometric feature model, and the image depth feature model have been described above. The steps performed by the cross-view alignment unit, the cross-target fusion unit, and the cross-scene thinker will be described in detail below, and will not be repeated here.

[0038] In an exemplary embodiment, determining the robot's execution action based on the feature set and the task instruction can be achieved through the following steps S302-S306:

[0039] Step S302: Determine a semantic fusion token set based on the task instruction, the semantic feature subset, and the three-dimensional space feature subset, wherein the semantic fusion token set includes: the semantic fusion token of the m-th view among the M views; the semantic fusion token of the m-th view is a token determined based on the semantic token that matches the task instruction in the semantic features of the m-th view and the three-dimensional space features of the m-th view, the semantic feature subset includes the semantic features of the m-th view, and the three-dimensional space feature subset includes the three-dimensional space features of the m-th view, where m is an integer greater than or equal to 1 and less than or equal to M;

[0040] It should be noted that the semantic fusion token set includes one or more semantic fusion tokens for each of the M perspectives. Among the M perspectives, the semantic features corresponding to the m-th perspective (e.g., perspective L) include multiple semantic tokens. One or more semantic tokens that match the task instruction are individually fused with the three-dimensional spatial features of the m-th perspective to obtain one or more semantic fusion tokens for the m-th perspective.

[0041] It should be noted that step S302 above can be performed using a cross-view aligner.

[0042] In the embodiment shown in step S302, through semantic filtering and three-dimensional feature fusion guided by task instructions, multi-view targets are accurately located and spatial information is injected under the condition of using only RGB images, which significantly improves the consistency and positional accuracy of target recognition, effectively alleviates the problems of occlusion and scale ambiguity, and improves the accuracy of action determination.

[0043] In an exemplary embodiment, determining a semantic fusion token set based on the task instruction, the semantic feature subset, and the three-dimensional spatial feature subset includes: modulating the semantic features of each viewpoint in the semantic feature subset to obtain a modulated semantic feature subset; determining a target semantic token set based on the modulated semantic feature subset and the task instruction, wherein the target semantic token set includes: the top K semantic tokens in the semantic features of the m-th viewpoint that have the highest semantic similarity to the task instruction; K is a positive integer less than the number of semantic tokens in the semantic features of the m-th viewpoint; and fusing each semantic token in the target semantic token set with the three-dimensional spatial features of the corresponding viewpoint to obtain the semantic fusion token set.

[0044] For example, such as Figure 3 As shown, the image semantic feature model in the cross-viewpoint aligner contains multiple semantic encoders, which are connected serially. It is assumed that the l-th semantic encoder outputs the semantic features of the m-th viewpoint. and will Modulation is performed using a characteristic linear modulation model to obtain modulated semantic features. :

[0045] ;

[0046] Where l represents the Transformer layer index in the semantic encoder. and These represent the scaling and bias parameters generated by the instruction embedding t, respectively. This represents element-wise multiplication. This represents the self-attention operation. The modulated semantic features are input into the (l+1)th layer semantic encoder, and the semantic features output from the (l+1)th layer semantic encoder are modulated using a feature linear modulation model. This process is repeated, with the modulated semantic features from the previous layer serving as the input to the next layer. Finally, the modulated semantic features output from the last layer of the semantic encoder are modulated to obtain a subset of the modulated semantic features. :

[0047] ;

[0048] Where L represents the last layer of the semantic encoder, the modulated semantic feature subset. Includes multiple semantic tokens, namely , This represents the number of semantic tokens in the m-th view. This represents the dimension of the semantic tokens. The similarity between each semantic token in the modulated semantic feature subset and the task instruction is calculated using a similarity calculation formula. Based on the similarity, the top K semantic tokens that best match the task instruction are determined, thus defining the target semantic token set. Combine the target semantic token set from the m-th viewpoint with the 3D spatial features from the m-th viewpoint. Perform single fusion to obtain the semantic fusion token set from the m-th perspective. :

[0049] ;

[0050] in, This represents a single fusion function, which is essentially based on... For query, with Perform cross-modal attention for keys and values ​​to inject 3D spatial information into the target token.

[0051] It should be noted that by modulating semantic features and filtering highly relevant tokens through task instructions, and then fusing three-dimensional spatial information, precise target focusing and spatial enhancement are achieved, which significantly improves the consistency of multi-view target recognition and the accuracy of three-dimensional positioning, and effectively suppresses background interference and occlusion ambiguity.

[0052] In an exemplary embodiment, determining a target semantic token set based on the modulated semantic feature subset and the task instruction includes: calculating the similarity between each semantic token in the modulated semantic feature subset and the task instruction using the following similarity calculation formula:

[0053] ;

[0054] in, Let be the similarity between the j-th semantic token in the semantic features of the i-th perspective and the task instruction. For the j-th semantic token, This involves embedding the encoded task instructions. A linear transformation matrix is ​​used to embed the instruction into a semantic comparison space; the target semantic token set is determined from the modulated semantic feature subset based on the similarity between each semantic token in the modulated semantic feature subset and the task instruction.

[0055] It should be noted that i is an integer greater than or equal to 1 and less than or equal to M.

[0056] For example, the semantic tokens are sorted according to their similarity, and the top K semantic tokens that best match the task instructions are used as the target tokens. , Let K represent the set of selected target token indices in the m-th view, and K represent the number of target tokens retained. Further, the set of target semantic tokens most relevant to the embedded instruction t in the m-th view is determined according to the following formula. :

[0057] ;

[0058] in, This represents an explicit semantic target selection function.

[0059] Step S304: Determine the target geometric relationship representation based on the geometric feature subset and the three-dimensional spatial feature subset, wherein the target geometric relationship representation is used to indicate the implicit aggregated representation of the spatial topology between multi-view targets;

[0060] For example, the geometric features corresponding to M viewpoints in the geometric feature subset can be spliced ​​together to obtain multi-view geometric features, and the three-dimensional spatial features corresponding to M viewpoints in the three-dimensional spatial feature subset can be spliced ​​together to obtain multi-view three-dimensional spatial features. Then, the geometric relationship representation of the target can be determined based on the multi-view geometric features and the multi-view three-dimensional spatial features.

[0061] It should be noted that step S304 above can be performed by the cross-target fusion processor.

[0062] In an exemplary embodiment, determining the target geometric relationship representation based on the geometric feature subset and the three-dimensional spatial feature subset can be achieved through the following steps S402-S406:

[0063] Step S402: Given that the geometric feature subset includes P geometric feature vectors output by P coding layers of the first encoder, and the three-dimensional space feature subset includes P three-dimensional space feature vectors output by P coding layers of the second encoder, determine P multi-view geometric features corresponding to the P geometric feature vectors and P multi-view three-dimensional space features corresponding to the P three-dimensional space feature vectors. The p-th multi-view geometric feature is obtained by concatenating the geometric features of M views in the p-th geometric feature vector, and the p-th multi-view three-dimensional space feature is obtained by concatenating the three-dimensional space features of M views in the p-th three-dimensional space feature vector. The first encoder is used to obtain the geometric feature subset based on the image data from the M views, and the second encoder is used to obtain the three-dimensional space feature subset based on the image data from the M views. P is an integer greater than or equal to 2, and p is an integer greater than or equal to 1 and less than or equal to P.

[0064] Optionally, such as Figure 3As shown, the image geometric feature model in the cross-target fusion generator contains multiple visual coding layers (i.e., the P coding layers of the first encoder mentioned above). These multiple visual coding layers are connected serially. Assuming there are P visual coding layers, each visual coding layer of the first encoder generates geometric features from multiple viewpoints. The multiple geometric features from the p-th layer are concatenated and merged to obtain multi-view geometric features. Similarly, the image depth feature model also contains multiple visual coding layers (i.e., the P coding layers of the second encoder mentioned above). Each visual coding layer of the second encoder generates three-dimensional spatial features from multiple viewpoints. The multiple three-dimensional spatial features of the p-th layer are then concatenated to obtain multi-view three-dimensional spatial features. .

[0065] Step S404: Based on P multi-view geometric features and P multi-view three-dimensional spatial features, obtain P fused features, wherein the p-th fused feature is the feature obtained by fusing the p-th multi-view geometric feature and the p-th multi-view three-dimensional spatial feature.

[0066] In an exemplary embodiment, step S404 above can be implemented as follows: P fusion features are obtained based on the P fusion weights, the P multi-view geometric features, and the P multi-view 3D spatial features, wherein the p-th fusion feature is a feature obtained by fusing the p-th multi-view geometric feature and the p-th multi-view 3D spatial feature based on the p-th fusion weight; wherein the first fusion weight among the P fusion weights is a first preset weight. The Pth fusion weight among the P fusion weights is the second preset weight. ;

[0067] Among the P fusion weights, the c-th fusion weight is... c is an integer greater than 1 and less than P.

[0068] It should be noted that determining P fusion weights using the above formula allows the weights to decay smoothly from shallow to deep layers.

[0069] For example, the multi-view geometric features of the p-th layer can be represented by the following formula. and multi-view three-dimensional spatial features Perform group fusion to obtain fusion features :

[0070]

[0071] in, Represents the group fusion function. This represents the fusion weights of the p-th layer visual encoder.

[0072] Step S406: Determine the target geometric relationship representation based on P fusion features.

[0073] In an exemplary embodiment, determining the target geometric relation representation based on the P fusion features includes: repeatedly performing the following operations until the target geometric relation representation is obtained, wherein, during the first operation, p equals 1, and the first geometric relation representation is a preset geometric relation representation; concatenating the p-th fusion feature and the p-th geometric relation representation among the P fusion features to obtain the p-th concatenated feature; performing block-level causal self-attention on the p-th concatenated feature to obtain the (p+1)-th geometric relation representation; determining the (p+1)-th geometric relation representation as the target geometric relation representation when p+1 equals P; and updating the value of p to p+1 when p+1 does not equal P.

[0074] To better understand, the following description uses an exemplary embodiment: The fused features of the first visual encoding layer and the first geometric relation representation are concatenated to obtain the concatenated features. Block-level causal self-attention is then applied to the concatenated features to obtain the second geometric relation representation. This second geometric relation representation is then concatenated with the fused features of the second visual encoding layer. Block-level causal self-attention is then applied to the concatenated features to obtain the third geometric relation representation. This process is repeated until the (P-1)th geometric relation representation is obtained. This (P-1)th geometric relation representation is then concatenated with the fused features of the (P-1)th visual encoding layer. Block-level causal self-attention is then applied to the concatenated features to obtain the Pth geometric relation representation, which is the target geometric relation representation. The process of determining the above geometric relationship can be summarized by the following formula:

[0075] ;

[0076] in, This indicates a feature concatenation operation. This indicates block-level causal self-attention. This represents the p-th geometric relation. This represents the geometric relationship of the (p+1)th layer. is the fusion feature of the p-th visual coding layer.

[0077] Step S306: Determine the robot's execution action based on the task instruction, the semantic fusion token set, and the target geometric relationship representation.

[0078] In an exemplary embodiment, determining the robot's execution action based on the task instruction, the semantic fusion token set, and the target geometric relation representation can be achieved through the following steps S502-S506:

[0079] Step S502: Determine a multi-view dynamic knowledge representation based on the task instruction and the semantic fusion token set, wherein the dynamic knowledge representation of the m-th view in the multi-view dynamic knowledge representation is a dynamic knowledge representation obtained after action reasoning based on the task instruction, the learnable dynamic token of the m-th view, and the semantic fusion token of the m-th view in the semantic fusion token set; the dynamic knowledge representation is used to indicate the semantic features of dynamic objects at future moments; the learnable dynamic token is an implicit state token used to learn semantic changes;

[0080] For example, such as Figure 6 As shown, in determining the set of semantic fusion tokens Target geometric relation representation After that, for each perspective Initialize the corresponding learnable dynamic token The dynamic knowledge of the m-th viewpoint is determined according to the following formula. :

[0081] ;

[0082] in, This represents the dynamic reasoning process in the cross-scenario thinker, where t is the instruction embedding after encoding the task instructions.

[0083] Step S504: Perform action reasoning based on the task instruction, the learnable depth token, and the target geometric relationship representation to obtain a deep knowledge representation, wherein the deep knowledge representation is used to indicate the depth features at future time moments, and the learnable depth token is an implicit structure token used to learn depth changes;

[0084] For example, initializing a learnable depth token And determine the deep knowledge representation according to the following formula. :

[0085] ;

[0086] in, .

[0087] Step S506: Determine the robot's execution action based on the semantic fusion token set, the target geometric relationship representation, the multi-view dynamic knowledge representation, and the deep knowledge representation.

[0088] It should be noted that by implicitly modeling future semantic evolution and geometric changes through learnable dynamic tokens and deep tokens, and combining this with spatiotemporal consistent attention, accurate reasoning about target dynamics and scene evolution can be achieved without explicit prediction.

[0089] It should be noted that the above steps S502-S506 can be executed through the cross-scenario thinker.

[0090] In an exemplary embodiment, determining the robot's execution actions based on the semantic fusion token set, the target geometric relationship representation, the multi-view dynamic knowledge representation, and the deep knowledge representation includes: inputting the semantic fusion token set, the target geometric relationship representation, the multi-view dynamic knowledge representation, the deep knowledge representation, and learnable action implicit tokens into an attention module based on multimodal spatiotemporal feature alignment to obtain a continuous action sequence, wherein the continuous action sequence has H action instruction information, the H action instruction information being used to instruct the robot's end effector to perform a continuous H-step action, and the h-th action instruction information including at least one of the following: an instruction for indicating the translation increment of the end effector, an instruction for indicating the rotation increment of the end effector, and an instruction for indicating the opening and closing of the gripper; H is an integer greater than or equal to 2, and h is an integer greater than or equal to 1 and less than or equal to H.

[0091] For example, a set of semantic fusion tokens Target geometric relation representation Multi-perspective dynamic knowledge representation Deep knowledge representation and learnable action implicit tokens The spatiotemporal consistency attention module (i.e., the attention module based on multimodal spatiotemporal feature alignment) inputs to the cross-scene thinker, and outputs action block A through the action head: ,in, This represents the spatiotemporal consistency attention module. Further, action block A is decoded from the action head into a continuous action sequence of length H, i.e. H represents the length of the action block. This represents the translation increment of the end effector at step h. This represents the rotational increment of the end effector at step h. This indicates the instruction for opening and closing the gripper in step h. Finally, the robot executes the continuous sequence of actions decoded from action block A.

[0092] In an exemplary embodiment, before determining a feature set based on the multi-view image data and determining the robot's execution action based on the feature set and the task instruction, the method further includes: acquiring sample training data, each piece of data in the sample training data including: a sample execution action, a sample task instruction corresponding to the sample execution action, sample image data from the M views, dynamic knowledge representation of the target view, and sample depth knowledge representation; using the sample training data to perform multiple rounds of training on an initial action prediction model until the value of the loss function of the initial action prediction model meets a preset condition, and determining the initial action prediction model that meets the preset condition as the target action prediction model, wherein the value of the loss function of the initial action prediction model is equal to the sum of a first loss value, a second loss value, and a third loss value; the first loss value is determined based on the predicted execution action and the sample execution action, the predicted... The execution action is the action predicted by the initial action prediction model based on the sample task instruction and the sample image data from the M perspectives; the second loss value is determined based on the predicted multi-view dynamic knowledge representation and the dynamic knowledge representation of the target perspective, wherein the predicted multi-view dynamic knowledge representation is the dynamic knowledge representation determined by the initial action prediction model based on the sample task instruction and the sample image data from the M perspectives, and the predicted multi-view dynamic knowledge representation includes the predicted dynamic knowledge representation of the M perspectives, and the dynamic knowledge representation is used to indicate the semantic features of dynamic objects at future times; the third loss value is determined based on the predicted depth knowledge representation and the sample depth knowledge representation, wherein the predicted depth knowledge representation is the depth knowledge representation determined by the initial action prediction model based on the sample task instruction and the sample image data from the M perspectives, and the depth knowledge representation is used to indicate the depth features at future times.

[0093] In an exemplary embodiment, the second loss value is determined by: determining the loss value between the dynamic knowledge representation of each view in the predicted multi-view dynamic knowledge representation and the dynamic knowledge representation of the target view, obtaining M first values; and determining the sum of the M first values ​​as the second loss value; the third loss value is determined by: determining the loss value between the deep knowledge representation of each view in the predicted deep knowledge representation and the deep knowledge representation of the corresponding view in the sample deep knowledge representation, obtaining M second values; and determining the sum of the M second values ​​as the third loss value.

[0094] To better understand the training process of the target action prediction model described above, the following explanation uses a specific example:

[0095] First, a sample training dataset is constructed, where each sample contains: the actual actions performed by the robot under a specific task (i.e., sample actions), the corresponding task instruction text, synchronized sample image data from M perspectives, and a dynamic knowledge representation of a preset target perspective and a sample deep knowledge representation of all perspectives. The dynamic knowledge representation is the ground truth value of the semantic feature evolution of the key target at a specified perspective in the future, and the deep knowledge representation is the ground truth value of the deep structure of the global scene in the future. This sample set is used to iteratively train the initial action prediction model for multiple rounds. Its overall loss function consists of a weighted combination of three parts:

[0096] First loss value The formula used to measure the difference between the action sequence predicted by the model and the actual sample action is calculated according to the following formula. Where A represents the actual action ultimately performed by the robot. The execution action predicted by the initial action prediction model;

[0097] Second loss value The method for supervising the dynamic evolution of the learning target in the model is as follows:

[0098] ,

[0099] in, To establish a fixed supervisory perspective, This represents the mask used to locate the target region. A dynamic knowledge representation under a fixed supervisory perspective. This represents the dynamic knowledge from the m-th perspective. This represents the mask used to locate the region corresponding to the m-th viewpoint;

[0100] Third loss value The ability of the constraint model to model global geometric evolution is calculated as follows:

[0101] ;

[0102] in, Let m be the deep knowledge representation from the m-th perspective. This represents deep knowledge from a fixed supervision perspective. The total loss of the initial action prediction model is obtained by summing the action prediction loss (first loss value), dynamic supervision loss (second loss value), and deep supervision loss (third loss value). : The initial motion prediction model is adjusted based on the total loss value.

[0103] To better understand the process of the above action determination method, the implementation flow of the above action determination method will be described below in conjunction with optional embodiments, but this is not intended to limit the technical solution of the embodiments of this application.

[0104] Figure 3 This is a schematic diagram of a cross-view aligner and a cross-target fusion unit for an action determination method according to an optional embodiment of this application, with reference to... Figure 3 The reasoning phase includes the following steps one through four:

[0105] Step 1: Determine the semantic features, set features, and 3D spatial features of each of the M viewpoints:

[0106] The initial motion prediction model acquires multi-view image observation data of the robot, I={0,L,R}, where 0 represents the main view, L represents the left view, and R represents the right view; let the RGB image of the m-th view be denoted as... The task instruction is q. Optionally, the task instruction q is encoded using a text encoder based on the SigLIP image semantic feature model to obtain the instruction embedding t: Then, the images from each viewpoint are encoded using image semantic feature models, image geometric feature models, and image depth feature models respectively, to obtain semantic features, geometric features, and 3D spatial features. These serve as the unified input for the subsequent cross-viewpoint aligner and cross-target fusion unit. The cross-viewpoint aligner is used to intelligently filter, semantically align, and geometrically enhance the visual features of the same task-related targets from multiple viewpoints, thereby constructing a cross-viewpoint consistent target representation. The cross-target fusion unit is used to integrate the geometric features and 3D structural information of multiple targets based on the target-level representation, and construct a globally consistent scene geometric understanding through layer-by-layer aggregation and causal reasoning.

[0107] Based on the semantic encoder of the SigLIP image semantic feature model, the RGB image at the m-th viewpoint Encode to obtain semantic features The visual encoder based on the image geometric feature model DINOv2 processes the RGB image at the m-th viewpoint. Encode to obtain geometric features The visual encoder of the image depth feature model VGGT processes the RGB image from the m-th viewpoint. Encode to obtain three-dimensional spatial features .

[0108] Step 2: Determine the set of semantic fusion tokens for each perspective using a cross-perspective aligner.

[0109] The image semantic feature model in the cross-view alignmentr contains multiple semantic encoders connected in series. It is assumed that the l-th semantic encoder outputs the semantic features of the m-th view. and will Modulation is performed using a characteristic linear modulation model to obtain modulated semantic features. The modulated semantic features are input into the (l+1)th layer semantic encoder, and the semantic features output from the (l+1)th layer semantic encoder are modulated using a feature linear modulation model. This process is repeated, with the modulated semantic features from the previous layer serving as the input to the next layer, ultimately resulting in a subset of modulated semantic features output from the last layer of the semantic encoder. .

[0110] Modulated semantic feature subset This involves multiple semantic tokens. The similarity between the j-th semantic token in the semantic features of the m-th viewpoint and the task instruction is determined using a similarity calculation formula. Based on the similarity, the top K semantic tokens that best match the task instruction are determined. Furthermore, based on these top K semantic tokens, the set of target semantic tokens most relevant to the embedded instruction t in the m-th viewpoint is determined. Combine the target semantic token set from the m-th viewpoint with the 3D spatial features from the m-th viewpoint. A single fusion is performed, and the semantic fusion token set under the m-th viewpoint is obtained after image deep feature model enhancement. It needs to be clarified that the above-mentioned three-dimensional spatial features are fused with the target semantic token set. It is the output of the last view encoder in the multi-layer visual encoder of the image depth feature model.

[0111] Step 3: Determine the geometric relationship representation of the target through a cross-target fusion engine:

[0112] The image geometric feature model in the cross-target fusion engine contains multiple layers of visual encoders, which are sequentially connected. Assuming there are l layers of visual encoders, each layer generates geometric features from multiple viewpoints. Multiple geometric features of the l-th layer By splicing and merging, multi-view geometric features are obtained. Similarly, image depth feature models also contain multiple layers of visual encoders, which are connected in series. For an l-layer visual encoder, each layer generates three-dimensional spatial features from multiple viewpoints. Multiple three-dimensional spatial features of the l-th layer By merging and combining the features, we can obtain multi-view three-dimensional spatial features. .

[0113] Multi-view geometric features of layer l and multi-view three-dimensional spatial features Perform group fusion to obtain fusion features Optionally, the fused features of the first visual encoding layer and the first geometric relation representation are concatenated to obtain the concatenated features. Block-level causal self-attention is then applied to the concatenated features to obtain the second geometric relation representation. This second geometric relation representation is then concatenated with the fused features of the first visual encoding layer, and block-level causal self-attention is applied to the concatenated features to obtain the second geometric relation representation. This process is repeated until the (P-1)th geometric relation representation is obtained. This (P-1)th geometric relation representation is then concatenated with the fused features of the (P-1)th visual encoding layer, and block-level causal self-attention is applied to the concatenated features to obtain the Pth geometric relation representation, which is the target geometric relation representation. .

[0114] Step 4: Determine the action to be performed based on the semantic fusion token set and the target geometric relation representation:

[0115] like Figure 4 As shown, in determining the set of semantic fusion tokens After that, for each perspective Initialize the corresponding learnable dynamic token Semantic fusion token set Learnable dynamic tokens and task instruction embedding The spatiotemporal consistency attention module of the cross-scenario thinker is input to obtain a multi-view dynamic knowledge representation. At the same time, such as Figure 5 As shown, in determining the target geometric relationship representation Next, initialize the learnable depth token. Represent the target geometric relationship Learnable deep tokens and task instruction embedding The input is fed into the spatiotemporal consistency attention module of the cross-scenario thinker to obtain a deep knowledge representation. .

[0116] like Figure 6 As shown, based on the semantic fusion token set Multi-perspective dynamic knowledge representation Target geometric relation representation Deep knowledge representation and learnable action implicit tokens The cross-scene thinker outputs action block A through the action head. Furthermore, action block A is decoded by the action head into a continuous action sequence of length H. Finally, the robot executes the continuous action sequence decoded from action block A.

[0117] It should be noted that during the training phase, the cross-scene thinker determines the action prediction loss (first loss value) of the initial action prediction model; the dynamic supervision loss (second loss value) is determined using only the dynamic prediction decoder present during training; and the deep supervision loss (third loss value) is determined using only the deep prediction decoder present during training. The action prediction loss (first loss value), dynamic supervision loss (second loss value), and deep supervision loss (third loss value) are then added together to obtain the total loss value of the initial action prediction model. The initial action prediction model is then adjusted based on this total loss value.

[0118] It should be noted that the above embodiments can be applied to indoor service robots. For example, when a dual-arm service robot performs a task in a kitchen or laboratory environment to grasp a target object from a table and place it at a designated location based on a verbal command, it acquires image observation data through a multi-view RGB camera and achieves semantic labeling consistency of the same target from different perspectives through a cross-view aligner. A stable global spatial geometric representation is constructed using a cross-target fusion processor, thereby accurately locating the target. During action execution, a cross-scene thinker utilizes pre-learned dynamic and depth tokens to infer the pose change trend of the target object and the evolution of the surrounding spatial structure during the robotic arm's grasping and handling process. This makes the action sequence forward-looking and continuous, effectively avoiding misgrabbing, collisions, or operation interruptions, significantly improving task completion rate and operational robustness. It can also be applied to mobile operation robots, such as performing tasks that involve moving and operating simultaneously in warehouse or laboratory environments, such as grasping moving targets or obstacle avoidance operations while walking.

[0119] To better understand, the following further explains the cross-view aligner, cross-target fusioner, and cross-scene thinker:

[0120] A cross-view aligner is used to select, correspond, and fuse features of the same task-related target from different viewpoints. Specifically, an image semantic feature model (e.g., SigLIP) extracts semantic features from multi-view images, and an explicit semantic object selection unit selects key features of multiple target groups from the semantic features of each viewpoint according to task instructions; an image depth feature model (e.g., VGGT) extracts geometric / depth-related features and provides cross-view correspondence and weighting information; a single-modal fusion unit fuses the features of the same target group from different viewpoints respectively, outputting a cross-view consistent target representation.

[0121] The cross-target fusion unit is used to further model the relationships between multiple targets based on the target-level representation. Specifically, the intermediate-layer visual features of the image geometric feature model (e.g., DINOv2) and the task aggregation token are input into the grouping fusion module and the implicit geometric relationship aggregation module, and then weighted layer by layer by the block weights provided by the image deep feature model, finally outputting a global scene geometric representation containing multi-target relationship information.

[0122] The future image and depth information prediction section consists of two parts. Figure 4 The connection between the cross-view aligner and the cross-scene thinker is illustrated. First, the cross-view aligner dynamically filters the local features within each viewpoint, retaining key tokens related to future changes and discarding static redundant tokens. Then, the learnable dynamic tokens are input into the cross-scene thinker to infer the future motion correlation between different targets and output the dynamic object image features at future moments. Figure 5 The connection between the cross-target fusion engine and the cross-scene thinker is described. First, the cross-target fusion engine constructs global geometric relationship representations between targets and between targets and the background, retaining global structural features effective for depth modeling while discarding redundant local appearance features. Then, after inference by the cross-scene thinker using learnable dynamic tokens, it outputs global depth features corresponding to each target at future time points. Thus, this part jointly completes the prediction from the current state to the future scene, providing a basis for action planning for long-term tasks.

[0123] The cross-scene thinker takes cross-perspective consistent target representation, global scene geometry representation, and task instructions as joint inputs, and introduces learnable dynamic tokens, learnable depth tokens, and learnable action tokens. It uses "spatiotemporally consistent attention" to uniformly model target semantics, scene geometry, and action intent. Specifically, the learnable dynamic token receives supervision from future dynamic objects during the training phase through a dynamic prediction decoder that exists only during training, to encode semantic knowledge related to future changes in the target. The learnable depth token receives supervision from future global depth through a depth prediction decoder that exists only during training, to encode geometric knowledge related to the future structure of the scene. The learnable action token comprehensively reads the above target-level and scene-level information under the unified attention framework, and outputs robotic arm action blocks through the action head. The action blocks include translation increments, rotation increments, and gripper opening and closing instructions of the end effector, thereby realizing end-to-end decision-making from current visual representation and task instructions to future spatiotemporal reasoning and action generation.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0125] This embodiment also provides an action determination device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0126] Figure 7 This is a structural block diagram of an action determining device according to an embodiment of this application, such as... Figure 7 As shown, the device includes:

[0127] The acquisition module 72 is used to acquire task instructions and multi-view image data, wherein the multi-view image data includes image data collected by the robot from M views, where M is an integer greater than or equal to 2;

[0128] The first determining module 74 is used to determine a feature set based on the multi-view image data, wherein the feature set includes a semantic feature subset, a geometric feature subset, and a three-dimensional spatial feature subset corresponding to the M views;

[0129] The second determining module 76 is used to determine the robot's execution action based on the feature set and the task instruction.

[0130] The aforementioned device acquires task instructions and multi-view image data collected by the robot from multiple perspectives. It extracts three-dimensional spatial features, semantic features, and geometric features from each perspective of the multi-view image data. Since three-dimensional spatial features can characterize the position of the target object in real three-dimensional space, and combined with semantic features and geometric features, it achieves cross-viewpoint and cross-modal spatial consistency perception. This enables action generation based on accurate three-dimensional spatial understanding, solves the problem of inaccurate robot action execution in related technologies, and improves the accuracy of action execution.

[0131] In an exemplary embodiment, the second determining module 76 is further configured to determine a semantic fusion token set based on the task instruction, the semantic feature subset, and the three-dimensional spatial feature subset, wherein the semantic fusion token set includes: the semantic fusion token of the m-th view among the M views; the semantic fusion token of the m-th view is a token determined based on the semantic token matching the task instruction in the semantic features of the m-th view and the three-dimensional spatial features of the m-th view, the semantic feature subset includes the semantic features of the m-th view, and the three-dimensional spatial feature subset includes the three-dimensional spatial features of the m-th view, where m is an integer greater than or equal to 1 and less than or equal to M; determine a target geometric relation representation based on the geometric feature subset and the three-dimensional spatial feature subset, wherein the target geometric relation representation is used to indicate an implicit aggregation representation of the spatial topology between multi-view targets; and determine the robot's execution action based on the task instruction, the semantic fusion token set, and the target geometric relation representation.

[0132] In an exemplary embodiment, the second determining module 76 is further configured to modulate the semantic features of each perspective in the semantic feature subset to obtain a modulated semantic feature subset; determine a target semantic token set based on the modulated semantic feature subset and the task instruction, wherein the target semantic token set includes: the top K semantic tokens in the semantic features of the m-th perspective that have the highest semantic similarity to the task instruction; K is a positive integer less than the number of semantic tokens in the semantic features of the m-th perspective; and fuse each semantic token in the target semantic token set with the three-dimensional spatial features of the corresponding perspective to obtain the semantic fusion token set.

[0133] In an exemplary embodiment, the second determining module 76 is further configured to calculate the similarity between each semantic token in the modulated semantic feature subset and the task instruction using the following similarity calculation formula: in, Let be the similarity between the j-th semantic token in the semantic features of the i-th perspective and the task instruction. For the j-th semantic token, This involves embedding the encoded task instructions. A linear transformation matrix is ​​used to embed the instruction into a semantic comparison space; the target semantic token set is determined from the modulated semantic feature subset based on the similarity between each semantic token in the modulated semantic feature subset and the task instruction.

[0134] In an exemplary embodiment, the second determining module 76 is further configured to, when the geometric feature subset includes P geometric feature vectors output by P coding layers of the first encoder and the three-dimensional space feature subset includes P three-dimensional space feature vectors output by P coding layers of the second encoder, determine P multi-view geometric features corresponding to the P geometric feature vectors and P multi-view three-dimensional space features corresponding to the P three-dimensional space feature vectors, wherein the p-th multi-view geometric feature is a feature obtained by concatenating the geometric features of the M views in the p-th geometric feature vector, and the p-th multi-view three-dimensional space feature is a feature obtained by concatenating the geometric features of the M views in the p-th geometric feature vector. The features are obtained by stitching together the three-dimensional spatial features from M perspectives; the first encoder is used to obtain the geometric feature subset based on the image data from the M perspectives, and the second encoder is used to obtain the three-dimensional spatial feature subset based on the image data from the M perspectives, where P is an integer greater than or equal to 2, and p is an integer greater than or equal to 1 and less than or equal to P; P fused features are obtained based on the P multi-view geometric features and the P multi-view three-dimensional spatial features, wherein the p-th fused feature is the feature obtained by fusing the p-th multi-view geometric feature and the p-th multi-view three-dimensional spatial feature; the target geometric relationship representation is determined based on the P fused features.

[0135] In an exemplary embodiment, the second determining module 76 is further configured to obtain P fusion features based on the P fusion weights, the P multi-view geometric features, and the P multi-view three-dimensional spatial features, wherein the p-th fusion feature is a feature obtained by fusing the p-th multi-view geometric feature and the p-th multi-view three-dimensional spatial feature based on the p-th fusion weight; wherein the first fusion weight among the P fusion weights is a first preset weight. The Pth fusion weight among the P fusion weights is the second preset weight. The c-th fusion weight among the P fusion weights c is an integer greater than 1 and less than P.

[0136] In an exemplary embodiment, the second determining module 76 is further configured to perform the following operations cyclically until the target geometric relation representation is obtained, wherein, during the first operation, p equals 1, and the first geometric relation representation is a preset geometric relation representation: concatenating the p-th fusion feature and the p-th geometric relation representation among the P fusion features to obtain the p-th concatenated feature; performing block-level causal self-attention on the p-th concatenated feature to obtain the p+1-th geometric relation representation; determining the p+1-th geometric relation representation as the target geometric relation representation when p+1 equals P; and updating the value of p to p+1 when p+1 does not equal P.

[0137] In an exemplary embodiment, the second determining module 76 is further configured to determine a multi-view dynamic knowledge representation based on the task instruction and the semantic fusion token set, wherein the dynamic knowledge representation of the m-th view in the multi-view dynamic knowledge representation is a dynamic knowledge representation obtained after action reasoning based on the task instruction, the learnable dynamic token of the m-th view, and the semantic fusion token of the m-th view in the semantic fusion token set; the dynamic knowledge representation is used to indicate the semantic features of dynamic objects at future times; the learnable dynamic token is an implicit state token used to learn semantic changes; and a depth knowledge representation is obtained by action reasoning based on the task instruction, the learnable depth token, and the target geometric relationship representation, wherein the depth knowledge representation is used to indicate the depth features at future times, and the learnable depth token is an implicit structure token used to learn depth changes; and the robot's execution action is determined based on the semantic fusion token set, the target geometric relationship representation, the multi-view dynamic knowledge representation, and the depth knowledge representation.

[0138] In an exemplary embodiment, the second determining module 76 is further configured to input the semantic fusion token set, the target geometric relationship representation, the multi-view dynamic knowledge representation, the deep knowledge representation, and the learnable action implicit tokens into an attention module based on multimodal spatiotemporal feature alignment to obtain a continuous action sequence, wherein the continuous action sequence has H action instruction information, the H action instruction information being used to instruct the robot's end effector to perform a continuous H-step action, and the h-th action instruction information including at least one of the following: an instruction for indicating the translation increment of the end effector, an instruction for indicating the rotation increment of the end effector, and an instruction for indicating the opening and closing of the gripper; H is an integer greater than or equal to 2, and h is an integer greater than or equal to 1 and less than or equal to H.

[0139] In an exemplary embodiment, the apparatus is further configured to: determine a feature set based on the multi-view image data using a target action prediction model; and determine the robot's execution action based on the feature set and the task instruction; before determining the feature set based on the multi-view image data and determining the robot's execution action based on the feature set and the task instruction, acquire sample training data, each piece of data in the sample training data including: a sample execution action, a sample task instruction corresponding to the sample execution action, sample image data from the M views, a dynamic knowledge representation of the target view, and a sample depth knowledge representation; use the sample training data to train an initial action prediction model for multiple rounds until the value of the loss function of the initial action prediction model satisfies a preset condition, and determine the initial action prediction model that satisfies the preset condition as the target action prediction model, wherein the value of the loss function of the initial action prediction model is equal to the sum of a first loss value, a second loss value, and a third loss value; The first loss value is determined based on the predicted execution action and the sample execution action, wherein the predicted execution action is the execution action predicted by the initial action prediction model based on the sample task instruction and the sample image data from the M perspectives; the second loss value is determined based on the predicted multi-view dynamic knowledge representation and the dynamic knowledge representation of the target perspective, wherein the predicted multi-view dynamic knowledge representation is the dynamic knowledge representation determined by the initial action prediction model based on the sample task instruction and the sample image data from the M perspectives, wherein the predicted multi-view dynamic knowledge representation includes the predicted dynamic knowledge representation of the M perspectives, and the dynamic knowledge representation is used to indicate the semantic features of dynamic objects at future times; the third loss value is determined based on the predicted depth knowledge representation and the sample depth knowledge representation, wherein the predicted depth knowledge representation is the depth knowledge representation determined by the initial action prediction model based on the sample task instruction and the sample image data from the M perspectives, and the depth knowledge representation is used to indicate the depth features at future times.

[0140] In an exemplary embodiment, the second loss value is determined by: determining the loss value between the dynamic knowledge representation of each view in the predicted multi-view dynamic knowledge representation and the dynamic knowledge representation of the target view, obtaining M first values; and determining the sum of the M first values ​​as the second loss value; the third loss value is determined by: determining the loss value between the deep knowledge representation of each view in the predicted deep knowledge representation and the deep knowledge representation of the corresponding view in the sample deep knowledge representation, obtaining M second values; and determining the sum of the M second values ​​as the third loss value.

[0141] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0142] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0143] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0144] Embodiments of this application also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0145] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0146] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0147] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods described in various embodiments of this application.

[0148] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the methods described in various embodiments of this application.

[0149] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0150] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A motion determination method characterized by comprising: include: Acquire task instructions and multi-view image data, wherein the multi-view image data includes image data collected by the robot from M perspectives, where M is an integer greater than or equal to 2; A feature set is determined based on the multi-view image data, wherein the feature set includes a semantic feature subset, a geometric feature subset, and a three-dimensional spatial feature subset corresponding to the M views; The robot's execution action is determined based on the feature set and the task instructions.

2. The method of claim 1, wherein, Determining the robot's execution actions based on the feature set and the task instructions includes: A semantic fusion token set is determined based on the task instruction, the semantic feature subset, and the three-dimensional spatial feature subset, wherein the semantic fusion token set includes: the semantic fusion token of the m-th view among the M views; the semantic fusion token of the m-th view is a token determined based on the semantic token that matches the task instruction in the semantic features of the m-th view and the three-dimensional spatial features of the m-th view, the semantic feature subset includes the semantic features of the m-th view, and the three-dimensional spatial feature subset includes the three-dimensional spatial features of the m-th view, where m is an integer greater than or equal to 1 and less than or equal to M; The target geometric relationship representation is determined based on the geometric feature subset and the three-dimensional spatial feature subset, wherein the target geometric relationship representation is used to indicate an implicit aggregated representation of the spatial topology between multi-view targets; The robot's execution action is determined based on the task instruction, the semantic fusion token set, and the target geometric relation representation.

3. The method of claim 2, wherein, The semantic fusion token set is determined based on the task instruction, the semantic feature subset, and the three-dimensional spatial feature subset, including: The semantic features of each perspective in the semantic feature subset are modulated to obtain the modulated semantic feature subset; A target semantic token set is determined based on the modulated semantic feature subset and the task instruction, wherein the target semantic token set includes: the top K semantic tokens in the semantic features of the m-th view that have the highest semantic similarity to the task instruction; K is a positive integer less than the number of semantic tokens in the semantic features of the m-th view; Each semantic token in the target semantic token set is fused with the three-dimensional spatial features of the corresponding viewpoint to obtain the semantic fusion token set.

4. The method of claim 3, wherein, Determining the target semantic token set based on the modulated semantic feature subset and the task instructions includes: The similarity between each semantic token in the modulated semantic feature subset and the task instruction is calculated using the following similarity calculation formula: wherein, a similarity between the jth semantic token in the semantic feature of the ith view and the task instruction, the jth semantic token, an instruction embedding after encoding the task instruction, a linear transformation matrix for mapping the instruction embedding to a semantic comparison space; The target semantic token set is determined from the modulated semantic feature subset based on the similarity between each semantic token in the modulated semantic feature subset and the task instruction.

5. The method of claim 2, wherein, Determining the target geometric relation representation based on the geometric feature subset and the three-dimensional spatial feature subset includes: When the geometric feature subset includes P geometric feature vectors output by P coding layers of the first encoder, and the three-dimensional space feature subset includes P three-dimensional space feature vectors output by P coding layers of the second encoder, P multi-view geometric features corresponding to the P geometric feature vectors and P multi-view three-dimensional space features corresponding to the P three-dimensional space feature vectors are determined. The p-th multi-view geometric feature is obtained by concatenating the geometric features of the M views in the p-th geometric feature vector, and the p-th multi-view three-dimensional space feature is obtained by concatenating the three-dimensional space features of the M views in the p-th three-dimensional space feature vector. The first encoder is used to obtain the geometric feature subset based on the image data of the M views, and the second encoder is used to obtain the three-dimensional space feature subset based on the image data of the M views. P is an integer greater than or equal to 2, and p is an integer greater than or equal to 1 and less than or equal to P. P fused features are obtained based on the P multi-view geometric features and the P multi-view three-dimensional spatial features, wherein the p-th fused feature is the feature obtained by fusing the p-th multi-view geometric feature and the p-th multi-view three-dimensional spatial feature; The target geometric relationship representation is determined based on the P fusion features.

6. The method according to claim 5, characterized in that, Based on the P multi-view geometric features and the P multi-view 3D spatial features, P fused features are obtained, including: P fusion features are obtained based on the P fusion weights, the P multi-view geometric features, and the P multi-view 3D spatial features, wherein the p-th fusion feature is the feature obtained by fusing the p-th multi-view geometric feature and the p-th multi-view 3D spatial feature based on the p-th fusion weight; Among them, the first fusion weight in the P fusion weights is the first preset weight. The Pth fusion weight among the P fusion weights is the second preset weight. The c-th fusion weight among the P fusion weights c is an integer greater than 1 and less than P.

7. The method according to claim 5, characterized in that, Determining the target geometric relationship representation based on the P fusion features includes: The following operations are performed repeatedly until the target geometric relation representation is obtained, wherein, during the first operation, p equals 1, and the first geometric relation representation is a preset geometric relation representation: The p-th fusion feature and the p-th geometric relationship representation among the P fusion features are concatenated to obtain the p-th concatenated feature; Perform block-level causal self-attention on the p-th concatenated feature to obtain the (p+1)-th geometric relation representation; When p+1 equals P, the p+1th geometric relation representation is determined as the target geometric relation representation; If p+1 is not equal to p, update the value of p to p+1.

8. The method according to claim 2, characterized in that, The robot's execution actions are determined based on the task instructions, the semantic fusion token set, and the target geometric relation representation, including: A multi-view dynamic knowledge representation is determined based on the task instruction and the semantic fusion token set, wherein the dynamic knowledge representation of the m-th view in the multi-view dynamic knowledge representation is a dynamic knowledge representation obtained after action reasoning based on the task instruction, the learnable dynamic token of the m-th view, and the semantic fusion token of the m-th view in the semantic fusion token set; the dynamic knowledge representation is used to indicate the semantic features of dynamic objects at future moments; the learnable dynamic token is an implicit state token used to learn semantic changes. Action reasoning is performed based on the task instructions, the learnable depth token, and the target geometric relationship representation to obtain a deep knowledge representation, wherein the deep knowledge representation is used to indicate the depth features at future moments, and the learnable depth token is an implicit structure token used to learn depth changes. The robot's execution actions are determined based on the semantic fusion token set, the target geometric relation representation, the multi-view dynamic knowledge representation, and the deep knowledge representation.

9. The method according to claim 8, characterized in that, The robot's execution actions are determined based on the semantic fusion token set, the target geometric relation representation, the multi-view dynamic knowledge representation, and the deep knowledge representation, including: The semantic fusion token set, the target geometric relationship representation, the multi-view dynamic knowledge representation, the deep knowledge representation, and the learnable action implicit tokens are input into an attention module based on multimodal spatiotemporal feature alignment to obtain a continuous action sequence. The continuous action sequence contains H action instruction information, which are used to instruct the robot's end effector to perform H consecutive steps of action. The h-th action instruction information includes at least one of the following: an instruction to indicate the translation increment of the end effector, an instruction to indicate the rotation increment of the end effector, or an instruction to indicate the opening and closing of the gripper. H is an integer greater than or equal to 2, and h is an integer greater than or equal to 1 and less than or equal to H.

10. The method according to claim 1, characterized in that, Determining a feature set based on the multi-view image data and determining the robot's execution action based on the feature set and the task instruction includes: determining a feature set based on the multi-view image data using a target action prediction model and determining the robot's execution action based on the feature set and the task instruction. Before determining the feature set based on the multi-view image data and determining the robot's execution action based on the feature set and the task instruction, the method further includes: acquiring sample training data, each piece of data in the sample training data including: sample execution action, sample task instruction corresponding to the sample execution action, sample image data of the M views, dynamic knowledge representation of the target view, and sample depth knowledge representation; using the sample training data to train the initial action prediction model in multiple rounds until the value of the loss function of the initial action prediction model meets a preset condition, and determining the initial action prediction model that meets the preset condition as the target action prediction model, wherein the value of the loss function of the initial action prediction model is equal to the sum of the first loss value, the second loss value, and the third loss value; The first loss value is determined based on the predicted execution action and the sample execution action, wherein the predicted execution action is the execution action predicted by the initial action prediction model based on the sample task instruction and the sample image data from the M perspectives; The second loss value is determined based on the predicted multi-view dynamic knowledge representation and the dynamic knowledge representation of the target view. The predicted multi-view dynamic knowledge representation is the dynamic knowledge representation determined by the initial action prediction model based on the sample task instruction and the sample image data of the M views. The predicted multi-view dynamic knowledge representation includes the predicted dynamic knowledge representation of the M views. The dynamic knowledge representation is used to indicate the semantic features of dynamic objects at future moments. The third loss value is determined based on the predicted depth knowledge representation and the sample depth knowledge representation. The predicted depth knowledge representation is the depth knowledge representation determined by the initial action prediction model based on the sample task instructions and the sample image data from the M perspectives. The depth knowledge representation is used to indicate the depth features at future moments.

11. The method according to claim 10, characterized in that, The second loss value is determined as follows: the loss value of the dynamic knowledge representation of each perspective in the predicted multi-view dynamic knowledge representation and the dynamic knowledge representation of the target perspective are determined to obtain M first values; the sum of the M first values ​​is determined as the second loss value. The third loss value is determined by: determining the loss value between the deep knowledge representation of each viewpoint in the predicted deep knowledge representation and the deep knowledge representation of the corresponding viewpoint in the sample deep knowledge representation, obtaining M second values; and determining the sum of the M second values ​​as the third loss value.

12. A motion determining device, characterized in that, include: The acquisition module is used to acquire task instructions and multi-view image data, wherein the multi-view image data includes image data collected by the robot from M views, where M is an integer greater than or equal to 2; The first determining module is used to determine a feature set based on the multi-view image data, wherein the feature set includes a semantic feature subset, a geometric feature subset, and a three-dimensional spatial feature subset corresponding to the M views; The second determining module is used to determine the robot's execution action based on the feature set and the task instruction.

13. A robot system, characterized in that, include: The robotic arm, the control terminal, and multiple image acquisition devices, wherein the control terminal is configured to perform the steps of the method described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 11.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 11.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 11.

Images