Action feature prediction method, model training method, task execution device, and medium

Abstract

The application provides an action feature prediction method, a model training method, a task execution device and a medium. The action feature prediction method comprises: obtaining a first noise visual feature and a first noise action feature of a current time step in a denoising process, wherein the first noise visual feature is used to predict surrounding environment information of the task execution device at a future time, and the first noise action feature is used to predict action information of the task execution device at the future time; determining association degree information between the first noise visual feature and the first noise action feature, wherein the association degree information is used to represent an influence degree between the first noise visual feature and the first noise action feature; performing attention calculation on the first noise visual feature and the first noise action feature based on the association degree information, or performing self-attention calculation on the first noise action feature, so as to obtain predicted action features. The technical scheme of the application can improve the prediction accuracy of the action features.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, specifically to a method for predicting action features, a model training method, a task execution device, and a medium. Background Technology

[0002] During the execution of a target task using a task execution device, the surrounding environment may change; for example, there may be moving objects nearby. To ensure the task execution device can complete the target task, its motion characteristics at future moments can be predicted. Existing motion characteristic prediction methods suffer from low accuracy. Summary of the Invention

[0003] In view of this, embodiments of this application provide a motion feature prediction method, a task execution device, and a medium, which can improve the prediction accuracy of motion features.

[0004] In a first aspect, embodiments of this application provide a motion feature prediction method, comprising: acquiring a first noise visual feature and a first noise motion feature at the current time step during denoising, wherein the first noise visual feature is used to predict the surrounding environment information of a task execution device at a future time, and the first noise motion feature is used to predict the motion information of the task execution device at a future time; determining correlation information between the first noise visual feature and the first noise motion feature, wherein the correlation information is used to characterize the degree of influence between the first noise visual feature and the first noise motion feature; performing bimodal interactive attention calculation on the first noise visual feature and the first noise motion feature based on the correlation information, or performing self-attention calculation on the first noise motion feature to obtain predicted motion features, wherein the predicted motion features are used to control the motion of the task execution device.

[0005] Secondly, embodiments of this application provide a model training method, comprising: acquiring a first training sample, wherein the first training sample includes a first target action noise feature, a first target visual noise feature, a first sample noise action feature, and a first sample noise visual feature; inputting first correlation information, the first sample noise action feature, and the first sample noise visual feature into a noise prediction sub-model to obtain predicted action noise features and predicted visual noise features, wherein the first correlation information is used to characterize the degree of influence between the first sample noise action feature and the first sample noise visual feature; determining a first loss value based on the difference between the predicted action noise feature and the first target action noise feature, and the difference between the predicted visual noise feature and the first target visual noise feature; and training the noise prediction sub-model based on the first loss value to obtain a trained noise prediction sub-model.

[0006] Thirdly, embodiments of this application provide a model training method, comprising: acquiring a second training sample, wherein the second training sample includes second sample noise action features and second sample noise visual features; inputting the second training sample into a correlation information discriminant network sub-model to obtain second correlation information, wherein the second correlation information is used to characterize the degree of influence between the second sample noise visual features and the second sample noise action features; determining a second loss value based on the second correlation information and target correlation information; and training the correlation information discriminant network sub-model based on the second loss value to obtain a trained correlation information discriminant network sub-model.

[0007] Fourthly, embodiments of this application provide a motion feature prediction device, comprising: an acquisition module, configured to acquire a first noise visual feature and a first noise motion feature at the current time step during denoising, wherein the first noise visual feature is used to predict the surrounding environment information of a task execution device at a future time, and the first noise motion feature is used to predict the motion information of the task execution device at a future time; a determination module, configured to determine the correlation information between the first noise visual feature and the first noise motion feature, wherein the correlation information is used to characterize the degree of influence between the first noise visual feature and the first noise motion feature; and a prediction module, configured to perform bimodal interactive attention calculation on the first noise visual feature and the first noise motion feature based on the correlation information, or perform self-attention calculation on the first noise motion feature, to obtain predicted motion features, wherein the predicted motion features are used to control the motion of the task execution device.

[0008] Fifthly, embodiments of this application provide a model training apparatus, comprising: an acquisition module for acquiring a first training sample, wherein the first training sample includes a first target action noise feature, a first target visual noise feature, a first sample noise action feature, and a first sample noise visual feature; a prediction module for inputting first correlation information, the first sample noise action feature, and the first sample noise visual feature into a noise prediction sub-model to obtain predicted action noise features and predicted visual noise features, wherein the first correlation information is used to characterize the degree of influence between the first sample noise action feature and the first sample noise visual feature; a determination module for determining a first loss value based on the difference between the predicted action noise feature and the first target action noise feature, and the difference between the predicted visual noise feature and the first target visual noise feature; and a training module for training the noise prediction sub-model based on the first loss value to obtain a trained noise prediction sub-model.

[0009] Sixthly, embodiments of this application provide a model training apparatus, comprising: an acquisition module for acquiring second training samples, wherein the second training samples include second sample noise action features and second sample noise visual features; a first determination module for inputting the second training samples into an association information discriminant network sub-model to obtain second association information, wherein the second association information is used to characterize the degree of influence between the second sample noise visual features and the second sample noise action features; a second determination module for determining a second loss value based on the second association information and target association information; and a training module for training the association information discriminant network sub-model based on the second loss value to obtain a trained association information discriminant network sub-model.

[0010] In a seventh aspect, embodiments of this application provide a task execution device, including a control module, which is used to execute the motion feature prediction method described in the first aspect above.

[0011] Eighthly, embodiments of this application provide an electronic device, including: a processor; and a memory for storing processor-executable instructions, wherein the processor is used to execute the action feature prediction method of the first aspect, the model training method of the second aspect, or the model training method of the third aspect described above.

[0012] Ninthly, embodiments of this application provide a computer-readable storage medium storing a computer program for executing the action feature prediction method of the first aspect, the model training method of the second aspect, or the model training method of the third aspect described above.

[0013] In a tenth aspect, embodiments of this application provide a computer program product, which includes a computer program. When the computer program is executed by a processor of a computer device, it enables the computer device to perform the motion feature prediction method of the first aspect, the model training method of the second aspect, or the model training method of the third aspect described above.

[0014] This application provides a motion feature prediction method, a model training method, a task execution device, and a medium. It determines whether to perform attention calculation only on the first noise motion feature or to perform bimodal interactive attention calculation on both based on the degree of influence between the first noise visual feature and the first noise motion feature. If the degree of influence between the first noise visual feature and the first noise motion feature is high, then bimodal interactive attention calculation is performed on both. This can more fully consider the influence between the information of the two modalities of motion and vision, improving the accuracy of predicted motion features and thus improving the accuracy of predicting future actions of the task execution device. Furthermore, this application does not use bimodal interactive attention calculation on the first noise visual feature and the first noise motion feature in all cases. This simplifies the calculation process to some extent, reduces the computational load, and shortens the computation time, thereby improving computational efficiency and thus improving the prediction efficiency of motion features. Attached Figure Description

[0015] Figure 1 The diagram shown is a schematic diagram of an action feature prediction scenario provided by an exemplary embodiment of this application.

[0016] Figure 2 The diagram shown is a flowchart illustrating an exemplary embodiment of the motion feature prediction method provided in this application.

[0017] Figure 3 The diagram shown is a schematic representation of the action feature prediction model provided in an exemplary embodiment of this application.

[0018] Figure 4 The diagram shown is a schematic representation of an exemplary embodiment of this application, illustrating the motion feature prediction process.

[0019] Figure 5 The diagram shown is a flowchart of an action feature prediction method provided in another exemplary embodiment of this application.

[0020] Figure 6 The diagram shown is a flowchart illustrating a model training method for a noise prediction sub-model provided in an exemplary embodiment of this application.

[0021] Figure 7 The diagram shown is a flowchart illustrating a model training method for a network sub-model for determining correlation information, provided in an exemplary embodiment of this application.

[0022] Figure 8 The diagram shown is a schematic diagram of the motion feature prediction device provided in an exemplary embodiment of this application.

[0023] Figure 9The diagram shown is a schematic diagram of the structure of a model training device provided in an exemplary embodiment of this application.

[0024] Figure 10 The diagram shown is a schematic diagram of the structure of a model training device provided in an exemplary embodiment of this application.

[0025] Figure 11 The diagram shown is a block diagram of an electronic device for performing an action feature prediction method or a model training method according to an exemplary embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] Application Overview

[0028] In the field of artificial intelligence, the embodied world model refers to an intelligent system framework that combines embodied cognition and world models. Based on environmental information of the task execution device, such as images or videos captured by sensors on the device, the embodied world model can predict the future action characteristics of the device. Based on the predicted action characteristics, the device can be controlled to perform the target task.

[0029] Specifically, in existing motion planning techniques, embodied world models can include generative visual models. Generative visual models (e.g., video diffusion models) are models capable of generating visual data (such as images and videos). These models can predict the surrounding environment of a task execution device at future moments based on images or videos of the device's environment at the current moment. The future images or videos may contain the task execution device's motion characteristics, or these characteristics can be obtained from the future images or videos. Based on these future motion characteristics, the task execution device's actions can be controlled to perform the target task.

[0030] However, existing motion feature prediction methods suffer from low accuracy.

[0031] To address the aforementioned technical problems, this application provides a method for predicting action features. By acquiring noise visual features and noise action features, determining the correlation information between them, and then performing bimodal interactive attention calculation on the noise visual features and noise action features based on this correlation information, or performing self-attention calculation on the noise action features, the predicted action features can be obtained. In this application embodiment, by first determining the correlation information between the noise visual features and noise action features, and then determining the specific method of subsequent calculation based on this correlation information, such as performing bimodal interactive attention calculation on the noise visual features and noise action features, or performing self-attention calculation on the noise action features, the accuracy of the predicted action features can be improved. Furthermore, by determining to perform self-attention calculation on the noise action features based on the correlation information, the calculation process can be simplified, the computational load reduced, and thus computational efficiency improved, i.e., the prediction efficiency of action features improved.

[0032] Exemplary scenario

[0033] The motion feature prediction method provided in this application can be applied to task execution devices, such as robots, vehicles, and drones, to perform target tasks. For example, a robot completes target tasks such as sweeping, cooking, walking a dog, watering flowers, serving food, or playing Go through a series of actions; a robot performs target tasks such as welding, assembly, and handling through a series of actions on a production line; a robot completes target tasks such as interception in a football match through a series of actions, such as intercepting a pass from an opponent's robot; and a drone completes its flight mission by adjusting its flight path to avoid collisions with obstacles.

[0034] The following example, using a robot as the task execution device and sweeping as the target task, illustrates the scenario of motion feature prediction. Figure 1 The diagram shown is a schematic representation of an action feature prediction scenario provided by an exemplary embodiment of this application. Figure 1 As shown, the motion feature prediction scenario includes a robot 100, which may be equipped with a motion feature prediction model. This model is used to predict the motion features of the robot 100 at future moments. The robot 100 can control its actions based on the predicted motion features at future moments. If the predicted motion at future moments is used to instruct the robot 100 to sweep the floor, then the robot 100 can sweep the floor based on the predicted motion at future moments.

[0035] It is understood that the robot 100 is equipped with a camera 110, which can be used to collect images or videos (image sequences) around the robot 100. The images or videos can be collectively referred to as visual data.

[0036] Specifically, robot 100 can acquire first noise visual features and first noise motion features at the current time step during the denoising process. The first noise visual features are used to predict the surrounding environment information of robot 100 at future times, and the first noise motion features are used to predict the motion information of robot 100 at future times. Further, robot 100 can determine the correlation information between the first noise visual features and the first noise motion features, wherein the correlation information characterizes the degree of influence between the first noise visual features and the first noise motion features. In addition, robot 100 can perform bimodal interactive attention calculation on the first noise visual features and the first noise motion features based on the correlation information, or perform self-attention calculation on the first noise motion features, to obtain predicted motion features. Based on the predicted motion features, robot 100 can control its own actions.

[0037] In one example, the action feature prediction model may include a correlation information discriminant network sub-model and a noise prediction sub-model. The action feature prediction model may determine the correlation information between a first noisy visual feature and a first noisy action feature based on the correlation information discriminant network sub-model.

[0038] Furthermore, the action feature prediction model can perform bimodal interactive attention calculation on the first noisy visual feature and the first noisy action feature based on the noise prediction sub-model to obtain predicted visual noise features and predicted action noise features. The action feature prediction model can then denoise the first noisy visual feature based on the predicted visual noise feature to obtain a second noisy visual feature, and denoise the first noisy action feature based on the predicted action noise feature to obtain a second noisy action feature. The second noisy visual feature can be updated with a new first noisy visual feature, and the second noisy action feature can be updated with a new first noisy action feature.

[0039] Optionally, the action feature prediction model can perform self-attention calculation on the first noisy action feature based on the noise prediction sub-model to obtain the predicted action noise feature. The action feature prediction model can then denoise the first noisy action feature based on the predicted action noise feature to obtain the third noisy action feature. The third noisy action feature can be updated to a new first noisy action feature.

[0040] The action feature prediction model can perform a repeated denoising process based on the updated first noisy visual features and the updated first noisy action features, or only based on the updated first noisy action features, until the denoising process is completed, to obtain the predicted action features. A detailed description of the denoising process can be found in the following description of the method embodiments.

[0041] In other examples, the action feature prediction method can be executed by an electronic device, such as a server, computer, tablet, or mobile phone. Furthermore, the electronic device may have an action feature prediction model deployed on it for executing the action feature prediction method. Furthermore, the electronic device may be communicatively connected to a task execution device to transmit the predicted action features to the task execution device.

[0042] It should be understood that the above application scenario examples are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited thereto. Rather, the embodiments of this application can be applied to any applicable scenario.

[0043] Exemplary methods

[0044] Figure 2 The diagram shown is a flowchart illustrating an exemplary embodiment of the motion feature prediction method provided in this application. Figure 2 The method can be executed by the task execution device, such as Figure 1 Robot 100 executes. (For example...) Figure 2 As shown, the action feature prediction method may include the following:

[0045] 210: Obtain the first noise visual feature and the first noise motion feature at the current time step during the denoising process.

[0046] The first noise visual feature is used to predict the surrounding environment information of the task execution device at future moments, and the first noise motion feature is used to predict the motion information of the task execution device at future moments.

[0047] In one example, the denoising process can be the process of denoising a first noisy visual feature and a first noisy motion feature to obtain denoised visual features and / or motion features. The denoised visual features can be predicted visual features, and similarly, the denoised motion features can be predicted motion features.

[0048] For example, a denoising process may include multiple time steps. In each time step, noise visual features and / or noise motion features may be denoised to a certain extent. The output of this time step can be used as the input for the next time step, and the input of the next time step is then denoised. This process is repeated, and the denoising process can be completed through multiple time steps. For ease of understanding, the noise visual features and noise motion features input to each time step can be referred to as the first noise visual feature and the first noise motion feature, respectively. The current time step can be used to represent the current step in the denoising process.

[0049] In one example, the first noisy visual feature and the first noisy action feature can be matrices, and their specific parameters can be set according to actual needs. Optionally, the first noisy visual feature and the first noisy action feature can be visual features and action features initialized with Gaussian noise, respectively. The visual features and action features initialized with Gaussian noise can simulate noise and uncertainty in the real environment, improve the robustness of the model, and enhance the predictive ability of the action.

[0050] Furthermore, in one example, the first noise action feature may include multiple noise action tokens, and the first noise visual feature may include multiple noise visual tokens.

[0051] 220: Determine the correlation information between the first noise visual feature and the first noise motion feature, wherein the correlation information is used to characterize the degree of influence between the first noise visual feature and the first noise motion feature.

[0052] Specifically, the correlation information is used not only to characterize the degree of influence between the first noise visual feature and the first noise motion feature, but also to indicate the attention calculation method of the task execution device on the first noise visual feature and the first noise motion feature during the denoising process, as detailed in step 230.

[0053] In one example, the correlation information can be represented by numbers, characters, images or other means, as long as it can reflect the degree of influence between the first noise visual feature and the first noise action feature.

[0054] 230: Perform bimodal interactive attention calculation on the visual features and motion features of the first noise based on correlation information, or perform self-attention calculation on the motion features of the first noise to obtain predicted motion features, wherein the predicted motion features are used to control the actions of the task execution device.

[0055] Specifically, if the determined correlation information indicates a high degree of influence between the first noise visual feature and the first noise action feature, then attention calculation for bimodal interaction is performed on the first noise visual feature and the first noise action feature based on the correlation information; if the determined correlation information indicates a low degree of influence between the first noise visual feature and the first noise action feature, then self-attention calculation can be performed on the first noise action feature.

[0056] In one example, the predicted motion features can characterize the actions of the task execution device at a future moment, and therefore the actions of the task execution device can be controlled based on these motion features.

[0057] This application provides a motion feature prediction method. It determines whether to perform attention calculation only on the first noise motion feature or to perform bimodal interactive attention calculation on both based on the degree of influence between the first noise visual feature and the first noise motion feature. If the degree of influence between the first noise visual feature and the first noise motion feature is high, then bimodal interactive attention calculation is performed on both. This can more fully consider the influence between the information of the two modalities of motion and vision, improving the accuracy of the predicted motion features and thus improving the accuracy of predicting the future actions of the task execution device. Furthermore, this application does not use bimodal interactive attention calculation on the first noise visual feature and the first noise motion feature in all cases. This simplifies the calculation process to some extent, reduces the computational load, and shortens the calculation time, thereby improving computational efficiency and thus improving the prediction efficiency of motion features.

[0058] According to one embodiment of this application, at each time step, the noise of a first noise visual feature / first noise motion feature can be predicted, i.e., visual noise feature / motion noise feature is obtained. Based on the predicted noise, i.e., visual noise feature / motion noise feature, the noise in the first noise visual feature / first noise motion feature is removed. In this way, the noise visual feature / noise motion feature can be gradually denoised through multiple time steps. For example, by denoising the noise motion feature through multiple time steps, motion features that characterize the motion information of the task execution device at future times can be obtained.

[0059] In one example, the process of performing bimodal interaction attention calculation on the first noise visual feature and the first noise action feature can be understood as the process of predicting the noise of the first noise visual feature and the noise of the first noise action feature; the process of performing self-attention calculation on the first noise action feature can be understood as the process of predicting the noise of the first noise action feature.

[0060] According to one embodiment of this application, a bimodal interactive attention calculation is performed on a first noise visual feature and a first noise action feature based on correlation information, or a self-attention calculation is performed on the first noise action feature to obtain a predicted action feature. This includes: when the correlation information is first information, performing bimodal interactive attention calculation on the first noise visual feature and the first noise action feature to obtain a second noise visual feature and a second noise action feature; and obtaining a predicted action feature based on the second noise visual feature and the second noise action feature; when the correlation information is second information, performing self-attention calculation on the first noise action feature to obtain a third noise action feature; and obtaining a predicted action feature based on the third noise action feature. The influence between the first noise visual feature and the first noise action feature represented by the second information is lower than the influence between the first noise visual feature and the first noise action feature represented by the first information.

[0061] Specifically, the first information and the second information can respectively represent the high and low degree of influence between the first noise visual feature and the first noise motion feature.

[0062] For example, if it is determined that the influence between the first noise visual feature and the first noise action feature is high, then the correlation information is determined as the first information. With the correlation information determined as the first information, bimodal interactive attention calculation is performed on the first noise visual feature and the first noise action feature to obtain the second noise visual feature and the second noise action feature. In one example, the second noise visual feature and the second noise action feature can be denoised features after the current time step. Based on the second noise visual feature and the second noise action feature, the predicted action feature can be obtained. In one example, the second noise visual feature and the second noise action feature can be updated to the first noise visual feature and the first noise action feature, respectively, and denoising is started in the next time step. This process is repeated, and the denoised action feature output in the last time step is the predicted action feature.

[0063] If the influence between the first noisy visual feature and the first noisy action feature is determined to be low, then the correlation information is determined as the second information. Given that the correlation information is determined to be the second information, self-attention is only performed on the first noisy action feature to obtain the third noisy action feature. In one example, the third noisy action feature can be the denoised feature after the current time step. Based on the third noisy action feature, the predicted action feature can be obtained. In another example, the third noisy action feature can be updated to the first noisy action feature, and denoising can begin at the next time step. At this time, the first noisy visual feature of the current time step can be input or not input at the next time step. This process is repeated, and the denoised action feature output at the last time step is the predicted action feature.

[0064] In one example, the relevance information can be represented by numbers; for example, the first piece of information can be 1 and the second piece of information can be 0, but it is not limited to this.

[0065] In this embodiment, when the correlation information is determined to be the first information, bimodal interactive attention calculation can be performed on the first noise visual feature and the first noise action feature. This combines information from both visual and action modalities, improving the accuracy of the predicted action feature. When the correlation information is determined to be the second information, self-attention calculation is performed only on the first noise action feature. This simplifies the calculation process to some extent, reduces the computational load, and shortens the calculation time, thereby improving computational efficiency and ultimately improving the prediction efficiency of the action feature.

[0066] According to an embodiment of this application, the denoising process includes T time steps, with the current time step being the t-th time step. Determining the correlation information between the first noise visual feature and the first noise motion feature includes: if the correlation information corresponding to the (t-1)-th time step is first information, determining the correlation information corresponding to the t-th time step based on the first noise visual feature and the first noise motion feature at the t-th time step, where t is an integer greater than 1; if the correlation information corresponding to the (t-1)-th time step is second information, determining the correlation information corresponding to the t-th time step as the second information.

[0067] In one example, T represents the total number of denoising steps in the denoising process, i.e., the total number of time steps in the denoising process. T controls the granularity of the denoising process; a larger T indicates that the denoising process is divided into more steps, with smaller noise variations at each step, resulting in a more refined denoising process and stronger predictive ability for future actions. A smaller T indicates a coarser denoising process, with larger noise variations at each step, resulting in weaker predictive ability for future actions.

[0068] In one example, T can be the default preset total number of time steps. T can be set randomly or according to certain rules, such as the function of the diffusion model itself. Furthermore, the value of t can range from 1 to T, i.e., t = 1, 2, 3, ..., T. For example, if T is 10, it means the denoising process has 10 time steps, and t takes values ​​of 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 in sequence.

[0069] In one example, if the correlation information of the previous time step is the first information, then the correlation information corresponding to the current time step can be determined based on the first noise visual features and the first noise motion features of the current time step. If the correlation information of the previous time step is the second information, then the correlation information can be directly determined as the second information, without needing to determine the correlation information corresponding to the current time step based on the first noise visual features and the first noise motion features of the current time step.

[0070] In one example, as the number of time steps increases, the influence between the first noisy visual feature and the first noisy action feature may gradually decrease. When the influence between the first noisy visual feature and the first noisy action feature is low, only the first noisy action feature can be denoised, while the denoising of the first noisy visual feature can be omitted. That is, the influence of the first noisy visual feature on the first noisy action feature can be ignored. This can improve prediction efficiency while ensuring the accuracy of the predicted action feature.

[0071] For example, if the correlation information at time step (t-1) is the second information, the correlation information at each time step from time step t to time step T can be determined as the second information. That is, from time step t to time step T, self-attention calculation can be performed only on the first noisy action feature. In one example, at each time step from time step t to time step T, self-attention calculation can be performed on the first noisy action feature to obtain the third noisy action feature. The third noisy action feature is updated to the first noisy action feature at the next time step. The next time step can perform self-attention calculation on the first noisy action feature to obtain the third noisy action feature. This process is repeated, and the denoised action feature output at the last time step is the predicted action feature.

[0072] In this embodiment, it is understood that the influence between the first noisy visual feature and the first noisy action feature gradually decreases as the denoising process progresses. When the correlation information is the second information, it indicates that the influence between the first noisy visual feature and the first noisy action feature at the current time step is low. Therefore, in subsequent time steps, the influence between the first noisy visual feature and the first noisy action feature will generally also be low. Thus, the correlation information for later time steps can be directly determined as the second information, which can, to a certain extent, ensure the prediction accuracy of action features, while simplifying the calculation process of correlation information, saving calculation time, and improving the prediction efficiency of action features. In summary, this embodiment can improve the prediction efficiency of action features while ensuring the prediction accuracy.

[0073] According to one embodiment of this application, determining the correlation information corresponding to time step t based on the first noise visual feature and the first noise action feature at time step t includes: inputting the first noise visual feature and the first noise action feature at time step t into a correlation information discrimination network sub-model to obtain the correlation information corresponding to time step t output by the correlation information discrimination network sub-model. The second noise visual feature and the first noise action feature are then subjected to bimodal interactive attention calculation to obtain the second noise visual feature and the second noise action feature. This includes: inputting the correlation information, the first noise visual feature, and the first noise action feature into a noise prediction sub-model to obtain the predicted visual noise feature and the predicted action noise feature output by the noise prediction sub-model; denoising the first noise visual feature based on the predicted visual noise feature to obtain the second noise visual feature; and denoising the first noise action feature based on the predicted action noise feature to obtain the second noise action feature. The correlation information discrimination network sub-model and the noise prediction sub-model are constructed based on a diffusion model.

[0074] Specifically, the correlation information can be obtained by relevant algorithms, models, or other methods. In this embodiment, the correlation information can be determined by the correlation information discrimination network sub-model. Visual noise features and motion noise features can be determined by the noise prediction sub-model. Visual noise features can characterize the noise predicted by the noise prediction sub-model from the first noise visual features. Similarly, motion noise features can characterize the noise predicted by the noise prediction sub-model from the first noise motion features.

[0075] In one example, when the correlation information corresponding to the (t-1)th time step is the first information, the first noise visual feature and the first noise action feature of the tth time step are input into the correlation information discrimination network sub-model to obtain the correlation information corresponding to the tth time step output by the correlation information discrimination network sub-model. The correlation information corresponding to the tth time step can be either the first information or the second information.

[0076] In one example, the correlation information discrimination network sub-model and the noise prediction sub-model can be sub-models in the action feature prediction model built based on the diffusion model. The action feature prediction model can have various structures, and one is described below as an example. Figure 3 The diagram shown is a schematic representation of the action feature prediction model provided in an exemplary embodiment of this application. Figure 3As shown, the action feature prediction model includes at least a correlation information discrimination network sub-model and a noise prediction sub-model. A1-t represents the first noisy action feature at time step t, V1-t represents the first noisy visual feature at time step t, pred-At represents the predicted action noise feature at time step t, pred-Vt represents the predicted visual noise feature at time step t, A2-t represents the second noisy action feature at time step t, i.e., the first noisy action feature at time step t+1, and V2-t represents the second noisy visual feature at time step t, i.e., the first noisy visual feature at time step t+1.

[0077] The correlation information discriminant network sub-model is used to determine the correlation information. The method for determining the correlation information has already been described above and will not be repeated here. In one example, the correlation information discriminant network sub-model inputs the correlation information, A1-t, and V1-t into the noise prediction sub-model. Given the predicted pred-Vt and pred-At output by the noise prediction sub-model, the action feature prediction model can denoise V1-t based on pred-Vt to obtain V2-t, and denoise A1-t based on pred-At to obtain A2-t.

[0078] In one example, the noise prediction sub-model may include an attention module that performs bimodal interaction attention calculation on V1-t and A1-t when the correlation information is the first information, and performs self-attention calculation on A1-t when the correlation information is the second information, to obtain a third noise action feature.

[0079] In one example, if the correlation information at time step t-1 is the second information, the correlation information discrimination network sub-model can record this second information. At time step t, the correlation information discrimination network sub-model can directly determine the correlation information as the second information. In this case, both A1-t and V1-t can be input into the correlation information discrimination network sub-model, but the sub-model will determine the correlation information as the second information. The noise prediction sub-model can then output the predicted pred-Vt and pred-At based on the correlation information, A1-t, and V1-t.

[0080] Optionally, if the correlation information at time step t-1 is the second information, the action feature prediction model or the noise prediction sub-model can record this second information. At time step t, the action feature prediction model or the noise prediction sub-model can directly determine the correlation information as the second information. At this time, the correlation information discrimination network sub-model can be paused, and A1-t can be directly input into the noise prediction sub-model. The noise prediction sub-model can output the predicted pred-At based on A1-t.

[0081] In one example, the noise prediction sub-model may also include modulation module 1, attention module, modulation module 2, and feedforward module.

[0082] According to one embodiment of this application, the denoising process includes T time steps, with the current time step being the t-th time step. The process of obtaining predicted action features based on second noise visual features and second noise action features includes: updating the second noise visual features and second noise action features to first noise visual features and first noise action features at the (t+1)-th time step, respectively; and obtaining predicted action features based on the correlation information corresponding to the (t+1)-th time step. The process of obtaining predicted action features based on third noise action features includes: updating the third noise action features to the first noise action features at the (t+1)-th time step; determining the correlation information corresponding to the (t+1)-th time step as second information; and obtaining predicted action features based on the third noise action features corresponding to the (t+1)-th time step. The third noise action features corresponding to the T-th time step are used as the predicted action features.

[0083] In one example, if the relevance information remains the first information, bimodal attention calculation is performed on the first noisy visual feature and the first noisy action feature to obtain the second noisy visual feature and the second noisy action feature. The second noisy visual feature and the second noisy action feature at time step t are updated to the first noisy visual feature and the first noisy action feature at time step t+1, respectively. Based on the relevance information corresponding to time step t+1, the first noisy visual feature and the first noisy action feature at time step t+1 are obtained. This denoising process is repeated until time step T, where the predicted action feature is obtained based on the second noisy action feature at time step T.

[0084] For example, continue with Figure 3 For example, Figure 3 As shown, when the correlation information is the first information, A2-t and V2-t at time step t are updated to A1-t and V1-t at time step t+1, respectively. That is, at time step t+1, A2-t and V2-t at time step t are used as the noise prediction sub-models at time step t+1. Based on the correlation information corresponding to time step t+1, A1-t and V1-t at time step t+1, V2-t and A2-t at time step t+1 are obtained. This process continues until time step T, where the predicted action features are obtained based on A2-t at time step T.

[0085] In one example, if the correlation information is the first information at some time steps, the above denoising process is repeated based on the updated first noise visual features and the updated first noise action features.

[0086] Furthermore, at other time steps, when the correlation information becomes the second information, self-attention calculation is performed on the first noisy action feature to obtain the third noisy action feature. The third noisy action feature is then updated to the first noisy action feature at time step t+1. The correlation information corresponding to time step t+1 is determined as the second information, and the corresponding third noisy action feature is obtained based on the first noisy action feature at time step t+1. This process is also a denoising process. This denoising process is repeated in this manner until time step T, at which the predicted action feature is obtained based on the third noisy action feature at time step T.

[0087] Furthermore, in one example, the action feature prediction model can perform a repeated denoising process based on the updated first noisy visual features and the updated first noisy action features when the correlation information is the first information, and perform a repeated denoising process based only on the updated first noisy action features when the correlation information is the second information, until the denoising process ends and the predicted action features are obtained.

[0088] For example, Figure 4 The diagram shown is a schematic representation of an exemplary embodiment of this application, illustrating a motion feature prediction process. Figure 4 As shown, A1-t represents the first noise motion feature at time step t, V1-t represents the first noise visual feature at time step t, pred-At represents the predicted motion noise feature at time step t, pred-Vt represents the predicted visual noise feature at time step t, A2-t represents the second noise motion feature at time step t, V2-t represents the second noise visual feature at time step t, and A3-t represents the third noise motion feature at time step t.

[0089] In the first time step, the correlation information discrimination network sub-model determines the correlation information as the first information based on A1-1 and V1-1. Then, the noise prediction sub-model obtains pred-A-1 and pred-V-1 for the first time step based on this first information, A1-1, and V1-1. Next, the action feature prediction model denoises A1-1 based on pred-A-1 to obtain A2-1, and the action feature prediction model denoises V1-1 based on pred-V-1 to obtain V2-1.

[0090] In the second time step, A2-1 and V2-1 from the first time step are used as A1-2 and V1-2 for the second time step, respectively. The correlation information discrimination network sub-model determines the correlation information as the first information based on A1-2 and V1-2. Then, the noise prediction sub-model obtains pred-A-2 and pred-V-2 for the second time step based on this first information, A1-2, and V1-2. Next, the action feature prediction model denoises A1-2 based on pred-A-2 to obtain A2-2, and denoises V1-2 based on pred-V-2 to obtain V2-2. Subsequent time steps follow the same pattern.

[0091] If, at time step m, A2-(m-1) and V2-(m-1) at time step (m-1) are taken as A1-m and V1-m at time step m, respectively, the correlation information discrimination network sub-model determines the correlation information as the second information based on A1-m and V1-m. For example, if the total number of time steps is 50, the correlation information can be 1 (first information) from time step 1 to time step 20, and 0 (second information) from time step 20 to time step 50, then m is 20. The noise prediction sub-model obtains pred-Am at time step m based on this second information, A1-m, and V1-m. Then, the action feature prediction model denoises A1-m based on pred-Am to obtain A3-m, which is the third noise action feature at time step m.

[0092] At time step m+1, the correlation information is still the second information. Therefore, A3-m at time step m is used as A1-(m+1) at time step m+1. The noise prediction sub-model obtains pred-A-(m+1) at time step m+1 based on this second information and A1-(m+1). Then, the action feature prediction model denoises A1-(m+1) based on pred-A-(m+1) to obtain A3-(m+1), which is the third noise action feature at time step m+1.

[0093] At time step T, the correlation information is still the second information. The noise prediction sub-model obtains the pred-AT at time step T based on this second information and A1-T. Then, the action feature prediction model denoises A1-T based on pred-AT to obtain A3-T, which is the third noisy action feature at time step T. Afterward, the action feature prediction model can obtain the predicted action features based on A3-T.

[0094] The depth of guidance for interaction can be dynamically adjusted based on the visual and action modalities and the difficulty of the samples, greatly optimizing inference efficiency.

[0095] In this embodiment, during the cyclic denoising process, denoising can be performed using information from different modalities based on the correlation information. This can, to a certain extent, ensure the prediction accuracy of action features. Furthermore, the guidance depth of the interaction between visual and action modal information can be dynamically adjusted using the correlation information. When the correlation information is used as secondary information, the calculation process of the correlation information and the denoising process of noisy action features can be simplified, improving the prediction efficiency of action features. In summary, this embodiment can improve the prediction efficiency of action features while ensuring their prediction accuracy.

[0096] According to one embodiment of this application, determining the correlation information corresponding to time step t based on the first noise visual feature and the first noise motion feature at time step t includes: determining the correlation information corresponding to time step t based on control conditions, the first noise visual feature and the first noise motion feature at time step t, wherein the control conditions include task text encoding and / or visual features, the task text encoding is used to characterize the task to be performed by the task execution device, and the visual features are used to characterize the current surrounding environment information of the task execution device.

[0097] Specifically, this control condition can be used to control the action feature prediction model to generate action features that satisfy the control condition.

[0098] In one example, the control condition may include a task text encoding, which represents the task text and describes the specific content of the task to be performed. Further, in another example, the task text can be encoded into a task text encoding using a text encoder. The task text encoding can be a text embedding vector, and the text encoder can be a text encoder based on a Contrastive Language-Image Pre-training (CLIP) architecture or other suitable encoder.

[0099] In one example, the control condition may include a visual latent code, which can include features encoded from the image or video data at the current moment. The visual latent code can be considered as objectively existing features, i.e., features without noise, which provide the model with a clear prediction direction and can improve the accuracy of the prediction results. Further, in one example, the image or video data at the current moment may be acquired by a camera of the task execution device. Specifically, the camera may acquire image or video data by capturing images or videos of the surrounding environment of the task execution device.

[0100] In one example, the control conditions may include task text encoding and visual latent code.

[0101] In this embodiment, the control conditions provide a clear prediction direction for determining the correlation information, which can improve the accuracy of determining the correlation information and thus improve the accuracy of the prediction results.

[0102] According to one embodiment of this application, a bimodal interactive attention calculation is performed on the first noise visual features and the first noise action features based on correlation information, or a self-attention calculation is performed on the first noise action features to obtain predicted action features. This includes: performing bimodal interactive attention calculation on the first noise visual features and the first noise action features based on control conditions and correlation information, or performing a self-attention calculation on the first noise action features to obtain predicted action features. The control conditions include task text encoding and / or visual features. The task text encoding is used to characterize the task performed by the task execution device, and the visual features are used to characterize the current surrounding environment information of the task execution device.

[0103] The control conditions have been described above and will not be repeated here.

[0104] In this embodiment, the control conditions provide a clear prediction direction for the prediction of action features, which can improve the accuracy of the prediction results.

[0105] Figure 5 The diagram shown is a flowchart of an action feature prediction method provided in another exemplary embodiment of this application. Figure 5 The example is Figure 2 Examples of the embodiments are provided below; to avoid repetition, the similarities can be referred to the descriptions in the above embodiments, and will not be repeated here. For example... Figure 5 As shown, the action feature prediction method may include the following steps.

[0106] 510: Obtain control conditions, the first noise action feature and the first noise visual feature at time step t.

[0107] t is an integer greater than or equal to 1, and t takes values ​​from 1 to T, where T represents the total number of time steps.

[0108] 520: Determine the correlation information at time step t, which is used to characterize the degree of influence between the first noise visual feature and the first noise motion feature.

[0109] 530: When the correlation information is the first information, perform bimodal interactive attention calculation on the first noise visual feature and the first noise action feature to obtain the second noise visual feature and the second noise action feature. Based on the second noise visual feature and the second noise action feature, obtain the predicted action feature. When the correlation information is the second information, perform self-attention calculation on the first noise action feature to obtain the third noise action feature. Based on the third noise action feature, obtain the predicted action feature.

[0110] The degree of influence between the first noise visual feature and the first noise motion feature in the second information representation is lower than the degree of influence between the first noise visual feature and the first noise motion feature in the first information representation.

[0111] 540: Determine if t has reached the total number of time steps.

[0112] If t has not reached the total number of time steps, proceed to step 550; otherwise, proceed to step 560.

[0113] 550: Take the second noise action feature and the second sound visual feature at time step t as the first noise action feature and the first noise visual feature at time step t+1, or take the third noise action feature at time step t as the first noise action feature at time step t+1, and set t to t+1.

[0114] Step 520 can be executed after step 550.

[0115] 560: The predicted action features are obtained based on the third noise action features at time step T.

[0116] According to one embodiment of this application, the attention calculation for the first noise visual feature and the first noise action feature at time step t can be achieved through a cross-attention mechanism, which may specifically include the following:

[0117] At time step t, attention is calculated based on the query vector group of the first noise visual feature (V1-t) and the value vector of the first noise action feature (A1-t) to obtain the predicted action noise feature (e.g., pred-At); attention is calculated based on the query vector group of the first noise action feature (A1-t) and the value vector of the first noise visual feature (V1-t) to obtain the predicted visual noise feature (e.g., pred-Vt).

[0118] Specifically, each modality can correspond to multiple query vectors and multiple value vectors.

[0119] In one example, the query vector represents the degree of attention the current element pays to other elements, while the value vector carries representation information for each element. Based on the query vector of the current element and the value vector of another element, the relationship between the current element and the other element can be calculated, thereby updating the representation vector of the other element. This process can be considered an attention calculation process.

[0120] In the above scheme, attention calculation in bimodal interaction can effectively capture the correlation between visual and action features, thereby improving the accuracy of action feature prediction.

[0121] In one example, self-attention calculation is performed on the first noisy action feature at time step t, which may specifically include the following:

[0122] At time step t, attention is calculated based on the query vector group of the first noise action feature (A1-t) and the value vector of the first noise action feature (A1-t) to obtain the predicted action noise feature (such as pred-At).

[0123] Before the task execution device can predict action features using the action feature prediction model, the model needs to be trained first. The training phase of the action feature prediction model will be described below. It is understandable that training the various sub-models separately can decompose a complex problem into multiple simpler ones, reducing training difficulty. The correlation information discrimination network sub-model and the noise prediction sub-model in the action feature prediction model will be trained separately below. It should be understood that the various stages of the model training process and their corresponding effects can be referenced from the relevant content in the action feature prediction method described above. The training process of the noise prediction sub-model will be introduced first.

[0124] Figure 6 The diagram shown is a schematic flowchart of a model training method for a noise prediction sub-model provided in an exemplary embodiment of this application. Figure 6 The method can be executed by training equipment, which can be a computer, server, etc. For example... Figure 6 As shown, the model training method may include the following steps.

[0125] 610: Obtain the first training sample, wherein the first training sample includes the first target action noise feature, the first target visual noise feature, the first sample noise action feature and the first sample noise visual feature.

[0126] Specifically, during the training phase, the first target action noise feature can be the true label of the action noise feature predicted by the noise prediction sub-model, and the first target visual noise feature can be the true label of the visual noise feature predicted by the noise prediction sub-model.

[0127] The first sample noise action features and the first sample noise visual features can be used as input samples for the noise prediction sub-model.

[0128] In one example, the first training sample can be generated based on a denoising diffusion probabilistic model sampler or other methods. It can be understood that the denoising diffusion probabilistic model (DDPM) sampler is a core component in diffusion models used to generate samples; it can generate the first training sample. For example, if the total number of time steps is 1000, the DDPM sampler can randomly sample the first training sample at time step t.

[0129] In one example, noise can be constructed based on the diffusion model principle, namely, constructing first target action noise features and first target visual noise features. Based on the first target action noise features and first target visual noise features, first sample noise action features and first sample noise visual features can be obtained. For example, adding noise to the initial action features using the first target action noise features yields the first sample noise action features, and adding noise to the initial visual features using the first target visual noise features yields the first sample noise visual features. The initial action features and initial visual features can be set according to actual needs.

[0130] 620: Input the first correlation information, the first sample noise action features and the first sample noise visual features into the noise prediction sub-model to obtain the predicted action noise features and the predicted visual noise features.

[0131] The first correlation information is used to characterize the degree of influence between the first sample noise action features and the first sample noise visual features.

[0132] For example, the noise prediction sub-model can be a diffusion transformer (DiT) model.

[0133] 630: Determine the first loss value based on the difference between the predicted motion noise features and the first target motion noise features, and the difference between the predicted visual noise features and the first target visual noise features.

[0134] For example, the first loss value can be calculated using Formula 1.

[0135] loss = ||pred_A_t – e_A_t|| 2 +λ||pred_V_t–e_V_t|| 2 …………Formula 1

[0136] Where loss represents the first loss value, pred_A_t represents the action noise feature predicted at time step t, pred_V_t represents the visual noise feature predicted at time step t, e_A_t represents the first target action noise feature at time step t, e_V_t represents the first target visual noise feature at time step t, and λ is a parameter, such as a hyperparameter.

[0137] It should be understood that in other examples, the first loss value can be calculated based on other formulas.

[0138] 640: The noise prediction sub-model is trained based on the first loss value to obtain the trained noise prediction sub-model.

[0139] In some embodiments, the network can be optimized through backpropagation, and the parameters of the noise prediction sub-model can be adjusted based on the first loss value to obtain the trained noise prediction sub-model.

[0140] This application provides a model training method. By inputting the first sample noise action features and the first sample noise visual features into a noise prediction sub-model, predicted action noise features and predicted visual noise features are obtained. Based on the differences between the predicted action noise features and predicted visual noise features and the first target action noise features and the first target visual noise features, a trained noise prediction sub-model is obtained. The noise prediction sub-model obtained by this model training method can utilize information from different modalities to denoise based on correlation information, thus ensuring the prediction accuracy of action features to a certain extent. Simultaneously, this can simplify the calculation process and improve the prediction efficiency of action features in some cases. In summary, this application can improve the prediction efficiency of action features while ensuring the prediction accuracy of future actions.

[0141] According to one embodiment of this application, when the denoising time step corresponding to the first sample noise action feature and the first sample noise visual feature is greater than the time step threshold, the first correlation information is first information; when the denoising time step corresponding to the first sample noise action feature and the first sample noise visual feature is less than or equal to the time step threshold, the first correlation information is second information or first information, wherein the degree of influence between the first sample noise action feature and the first sample noise visual feature represented by the second information is lower than the degree of influence between the first sample noise action feature and the first sample noise visual feature represented by the first information.

[0142] Specifically, the first correlation information, the first sample noise action features, and the first sample noise visual features are all used as inputs to the noise prediction sub-model.

[0143] In one example, if the denoising time step is greater than the time step threshold, it indicates that the noise in the first sample noisy action features and the first sample noisy visual features is significant, and they need to reference each other to improve denoising quality. Therefore, the first correlation information can be set as the first information in this case. If the denoising time step is less than or equal to the time step threshold, it indicates that the noise in the first sample noisy action features and the first sample noisy visual features is not particularly significant, and it is unclear whether they need to reference each other to improve denoising quality. In this case, the first correlation information can be randomly set as either the first information or the second information to allow the model to learn on its own.

[0144] It should be understood that the time step threshold can be set as needed or based on experience.

[0145] In this embodiment, by setting the first correlation information according to the time step threshold, the training effect of the noise prediction sub-model can be improved. Furthermore, since the time step threshold can be set as needed or empirically, the flexibility of the noise prediction sub-model training process can be increased.

[0146] Next, we will introduce Figure 3 The training process of the sub-model of the correlation information discrimination network.

[0147] Figure 7 The diagram shown is a schematic representation of the training process of a sub-model for a correlation information discrimination network provided in an exemplary embodiment of this application. Figure 7 The method can be executed by training equipment, which can be a computer, server, etc. For example... Figure 7 As shown, the training process includes the following steps.

[0148] 710: Obtain the second training sample, wherein the second training sample includes the second sample noise action features and the second sample noise visual features.

[0149] Specifically, the second training sample serves as the input sample for the correlation information discrimination network sub-model. By using the second training sample as the input sample for the correlation information discrimination network sub-model, the trained correlation information discrimination network sub-model can determine correlation information based on noise action features and noise visual features.

[0150] The construction process of the second sample noise action features and the second sample noise visual features is similar to the construction process of the first sample noise action features and the first sample noise visual features described above. To avoid repetition, it will not be described again here.

[0151] 720: Input the second training sample into the correlation information discriminant network sub-model to obtain the second correlation information, wherein the second correlation information is used to characterize the degree of influence between the second sample noise visual features and the second sample noise action features.

[0152] The second correlation information can be either the first information or the second information. The first and second information have been described in the previous text and will not be repeated here.

[0153] 730: Determine the second loss value based on the second correlation information and the target correlation information.

[0154] Specifically, the target correlation information is the true label of the second correlation information output by the correlation information discrimination network sub-model.

[0155] In some embodiments, a second loss value can be calculated using cross-entropy loss. For example, the second loss value can be determined using Equation 2.

[0156] l_p = CrossEntropy(P_i,G_i)……………Formula 2

[0157] Where l_p represents the second loss value, P_i represents the second correlation information output by the correlation information discrimination network sub-model, and G_i represents the target correlation information.

[0158] Target relevance information can be set manually or according to certain rules.

[0159] 740: The correlation information discrimination network sub-model is trained based on the second loss value to obtain the trained correlation information discrimination network sub-model.

[0160] In some embodiments, the parameters of the correlation information discrimination network sub-model are adjusted based on the second loss value to obtain the trained correlation information discrimination network sub-model.

[0161] This application provides a model training method. By inputting the noise action features and noise visual features of a second sample into a correlation information discriminant network sub-model, predicted second correlation information is obtained. Based on the difference between the predicted second correlation information and the target correlation information, the trained correlation information discriminant network sub-model is obtained. The correlation information discriminant network sub-model obtained by this model training method has the ability to determine correlation information based on noise action features and noise visual features. Because the correlation information determined by the correlation information discriminant network sub-model is relatively high, the noise prediction sub-model can use information from different modalities to denoise based on the more accurate correlation information. This can, to a certain extent, ensure the prediction accuracy of action features.

[0162] According to one embodiment of this application, a correlation information discrimination network sub-model can be trained based on a pre-trained noise prediction sub-model. For example, the pre-trained noise prediction sub-model can be used to determine the target correlation information.

[0163] According to one embodiment of this application, a second training sample and first information are input into a trained noise prediction sub-model to obtain a predicted first feature, wherein the predicted first feature includes a predicted first action noise feature and a predicted first visual noise feature; a third loss value is determined based on the difference between the predicted first action noise feature and the second target action noise feature, and the difference between the predicted first visual noise feature and the second target visual noise feature; the second training sample and second information are input into the trained noise prediction sub-model to obtain a predicted second feature, wherein the predicted second feature includes a predicted second action noise feature and a predicted second visual noise feature; a fourth loss value is determined based on the difference between the predicted second action noise feature and the second target action noise feature, and the difference between the predicted second visual noise feature and the second target visual noise feature; if the third loss value is less than the fourth loss value, the target correlation information is determined as the first information; if the third loss value is greater than the fourth loss value, the target correlation information is determined as the second information.

[0164] The second target motion noise feature and the second target visual noise feature are similar to the first target motion noise feature and the first target visual noise feature mentioned above. The construction process and related content of the second target motion noise feature and the second target visual noise feature can be found in the construction process and related content of the first target motion noise feature and the first target visual noise feature mentioned above. To avoid repetition, they will not be repeated here.

[0165] For example, in some embodiments, the pseudo-true value is the real label of the model training, and can be used as target correlation information. Specifically, the second training sample and the first information (e.g., 1) can be input into the noise prediction sub-model, and the third loss value can be calculated according to Formula 1 above; the second training sample and the second information (e.g., 0) can be input into the noise prediction sub-model, and the fourth loss value can be calculated according to Formula 1 above. The pseudo-true value is determined based on the third loss value and the fourth loss value, and is determined as the target correlation information.

[0166] For example, the pseudo-true value can be calculated using Formula 3.

[0167] G_i=(l_0–l_1)>l_d……………Formula 3

[0168] Where G_i represents the pseudo-true value, l_0 represents the fourth loss value, l_1 represents the third loss value, and l_d represents the parameter, which can be preset as needed.

[0169] The meaning of Formula 3 above is as follows: If the difference between l_0 and l_1 is greater than the threshold l_d, it means that the loss corresponding to the input of the second information (e.g., 0) is large and the loss corresponding to the input of the first information (e.g., 1) is small, and the Gi value is 1, that is, the target correlation information is 1 at this time; If the difference between l_0 and l_1 is less than the threshold l_d, it means that the loss corresponding to the input of the first information (e.g., 1) is large and the loss corresponding to the input of the second information (e.g., 0) is small, and the Gi value is 0, that is, the target correlation information is 0 at this time.

[0170] In this embodiment, by utilizing the trained noise prediction sub-model to determine the target correlation information, the correlation information discrimination network sub-model can be equipped with the ability to determine the interaction depth of both action and visual modal information, i.e., the ability to determine correlation information. The model training method provided in this application embodiment can obtain both the noise prediction sub-model and the correlation information discrimination network sub-model. Based on the correlation information discrimination network sub-model, the interaction depth of action and visual information can be determined, while based on the noise prediction sub-model, action features and visual features can be predicted.

[0171] According to one embodiment of this application, the first correlation information, the first sample noise action feature, and the first sample noise visual feature are input into the noise prediction sub-model to obtain the predicted action noise feature and the predicted visual noise feature. This includes: inputting control conditions, the first correlation information, the first sample noise action feature, and the first sample noise visual feature into the noise prediction sub-model to obtain the predicted action noise feature and the predicted visual noise feature. The control conditions include task text encoding and / or visual features. The task text encoding is used to characterize the task performed by the task execution device, and the visual features are used to characterize the current surrounding environment information of the task execution device.

[0172] In this embodiment, the control conditions provide a clear prediction direction for determining the correlation information. Using the control conditions as input samples for training enables the trained correlation information discrimination network sub-model to determine the correlation information based on the clear prediction direction, thereby improving the accuracy of the correlation information determination and thus improving the accuracy of the prediction results.

[0173] Similarly, control conditions can be added during the training of the noise prediction sub-model.

[0174] Exemplary device

[0175] Figure 8 The diagram shown is a schematic representation of the motion feature prediction device provided in an exemplary embodiment of this application. Figure 8 As shown, the motion feature prediction device 800 includes: an acquisition module 810, a determination module 820, and a prediction module 830.

[0176] The acquisition module 810 is used to acquire the first noise visual feature and the first noise motion feature at the current time step during the denoising process. The first noise visual feature is used to predict the surrounding environment information of the task execution device at future times, and the first noise motion feature is used to predict the motion information of the task execution device at future times. The determination module 820 is used to determine the correlation information between the first noise visual feature and the first noise motion feature. The correlation information is used to characterize the degree of influence between the first noise visual feature and the first noise motion feature. The prediction module 830 is used to perform bimodal interactive attention calculation on the first noise visual feature and the first noise motion feature based on the correlation information, or to perform self-attention calculation on the first noise motion feature, to obtain the predicted motion feature. The predicted motion feature is used to control the action of the task execution device.

[0177] This application provides a motion feature prediction device. It determines whether to perform attention calculation only on the first noise motion feature or to perform bimodal interactive attention calculation on both based on the degree of influence between the first noise visual feature and the first noise motion feature. If the degree of influence between the first noise visual feature and the first noise motion feature is high, then bimodal interactive attention calculation is performed on both. This can more fully consider the influence between the information of the two modalities of motion and vision, improving the accuracy of the predicted motion features and thus improving the accuracy of predicting the future actions of the task execution device. Furthermore, this application does not use bimodal interactive attention calculation on the first noise visual feature and the first noise motion feature in all cases. This simplifies the calculation process to some extent, reduces the computational load, and shortens the calculation time, thereby improving computational efficiency and thus improving the prediction efficiency of motion features.

[0178] According to an embodiment of this application, the prediction module 830 is configured to: when the correlation information is the first information, perform bimodal interactive attention calculation on the first noise visual feature and the first noise action feature to obtain the second noise visual feature and the second noise action feature, and obtain the predicted action feature based on the second noise visual feature and the second noise action feature; when the correlation information is the second information, perform self-attention calculation on the first noise action feature to obtain the third noise action feature, and obtain the predicted action feature based on the third noise action feature, wherein the degree of influence between the first noise visual feature and the first noise action feature represented by the second information is lower than the degree of influence between the first noise visual feature and the first noise action feature represented by the first information.

[0179] According to an embodiment of this application, the denoising process includes T time steps, with the current time step being the t-th time step. The determining module 820 is configured to: determine the correlation information corresponding to the t-th time step based on the first noise visual feature and the first noise action feature when the correlation information corresponding to the (t-1)-th time step is the first information, where t is an integer greater than 1; and determine the correlation information corresponding to the t-th time step as the second information when the correlation information corresponding to the (t-1)-th time step is the second information.

[0180] According to one embodiment of this application, the determining module 820 is used to input the first noise visual feature and the first noise action feature at time step t into the correlation information discrimination network sub-model to obtain the correlation information corresponding to time step t output by the correlation information discrimination network sub-model. Further, in one example, the prediction module 830 is used to: input the correlation information, the first noise visual feature, and the first noise action feature into the noise prediction sub-model to obtain the predicted visual noise feature and the predicted action noise feature output by the noise prediction sub-model; perform denoising processing on the first noise visual feature based on the predicted visual noise feature to obtain the second noise visual feature; and perform denoising processing on the first noise action feature based on the predicted action noise feature to obtain the second noise action feature, wherein the correlation information discrimination network sub-model and the noise prediction sub-model are constructed based on a diffusion model.

[0181] According to an embodiment of this application, the denoising process includes T time steps, with the current time step being the t-th time step. The prediction module 830 is used to: update the second noise visual feature and the second noise action feature to the first noise visual feature and the first noise action feature at the t+1-th time step, respectively; and obtain the predicted action feature based on the correlation information corresponding to the t+1-th time step. Obtaining the predicted action feature based on the third noise action feature includes: updating the third noise action feature to the first noise action feature at the t+1-th time step; determining the correlation information corresponding to the t+1-th time step as second information; and obtaining the predicted action feature based on the third noise action feature corresponding to the t+1-th time step. The third noise action feature corresponding to the T-th time step is used as the predicted action feature.

[0182] According to an embodiment of this application, the determining module 820 is used to determine the correlation information corresponding to the time step t based on control conditions, the first noise visual features and the first noise motion features at the time step t. The control conditions include task text encoding and / or visual features. The task text encoding is used to characterize the task to be performed by the task execution device, and the visual features are used to characterize the current surrounding environment information of the task execution device.

[0183] According to one embodiment of this application, the prediction module 830 is used to perform bimodal interactive attention calculation on the first noise visual features and the first noise action features based on control conditions and correlation information, or to perform self-attention calculation on the first noise action features to obtain predicted action features. The control conditions include task text encoding and / or visual features. The task text encoding is used to characterize the task to be performed by the task execution device, and the visual features are used to characterize the current surrounding environment information of the task execution device.

[0184] It should be understood that the operation and function of the acquisition module 810, the determination module 820, and the prediction module 830 in the above embodiments can be referred to the above. Figure 2 or Figure 5 The description of the action feature prediction method provided in the embodiments will not be repeated here to avoid repetition.

[0185] Figure 9 The diagram shown is a schematic representation of the structure of a model training apparatus provided in an exemplary embodiment of this application. Figure 9 As shown, the model training device 900 includes: an acquisition module 910, a prediction module 920, a determination module 930, and a training module 940.

[0186] The acquisition module 910 is used to acquire a first training sample, wherein the first training sample includes a first target action noise feature, a first target visual noise feature, a first sample noise action feature, and a first sample noise visual feature. The prediction module 920 is used to input the first correlation information, the first sample noise action feature, and the first sample noise visual feature into the noise prediction sub-model to obtain predicted action noise features and predicted visual noise features, wherein the first correlation information is used to characterize the degree of influence between the first sample noise action feature and the first sample noise visual feature. The determination module 930 is used to determine a first loss value based on the difference between the predicted action noise feature and the first target action noise feature, and the difference between the predicted visual noise feature and the first target visual noise feature. The training module 940 is used to train the noise prediction sub-model based on the first loss value to obtain the trained noise prediction sub-model.

[0187] This application provides a model training apparatus. By inputting the first sample noise action features and the first sample noise visual features into a noise prediction sub-model, predicted action noise features and predicted visual noise features are obtained. Based on the differences between the predicted action noise features and predicted visual noise features and the first target action noise features and the first target visual noise features, a trained noise prediction sub-model is obtained. The noise prediction sub-model obtained by this model training method can utilize information from different modalities to denoise based on correlation information, thus ensuring the prediction accuracy of action features to a certain extent. Simultaneously, this simplifies the calculation process in some cases and improves the prediction efficiency of action features. In summary, this application embodiment can improve the prediction efficiency of action features while ensuring the prediction accuracy of future actions.

[0188] According to one embodiment of this application, when the denoising time step corresponding to the first sample noise action feature and the first sample noise visual feature is greater than the time step threshold, the first correlation information is first information; when the denoising time step corresponding to the first sample noise action feature and the first sample noise visual feature is less than or equal to the time step threshold, the first correlation information is second information or first information, wherein the degree of influence between the first sample noise action feature and the first sample noise visual feature represented by the second information is lower than the degree of influence between the first sample noise action feature and the first sample noise visual feature represented by the first information.

[0189] According to one embodiment of this application, the acquisition module 910 is further configured to: acquire a second training sample, wherein the second training sample includes second sample noise action features and second sample noise visual features; input the second training sample into the correlation information discrimination network sub-model to obtain second correlation information, wherein the second correlation information is used to characterize the degree of influence between the second sample noise visual features and the second sample noise action features. The determination module 930 is further configured to: determine a second loss value based on the second correlation information and the target correlation information. The training module 940 is further configured to: train the correlation information discrimination network sub-model based on the second loss value to obtain the trained correlation information discrimination network sub-model.

[0190] According to one embodiment of this application, the prediction module 920 is further configured to: input the second training sample and the first information into the trained noise prediction sub-model to obtain a predicted first feature, wherein the predicted first feature includes a predicted first action noise feature and a predicted first visual noise feature. The determination module 930 is further configured to: determine a third loss value based on the difference between the predicted first action noise feature and the second target action noise feature, and the difference between the predicted first visual noise feature and the second target visual noise feature. The prediction module 920 is further configured to: input the second training sample and the second information into the trained noise prediction sub-model to obtain a predicted second feature, wherein the predicted second feature includes a predicted second action noise feature and a predicted second visual noise feature. The determination module 930 is further configured to: determine a fourth loss value based on the difference between the predicted second action noise feature and the second target action noise feature, and the difference between the predicted second visual noise feature and the second target visual noise feature; if the third loss value is less than the fourth loss value, determine the target correlation information as the first information; if the third loss value is greater than the fourth loss value, determine the target correlation information as the second information.

[0191] According to an embodiment of this application, the prediction module 920 is used to: input control conditions, first correlation information, first sample noise action features and first sample noise visual features into the noise prediction sub-model to obtain predicted action noise features and predicted visual noise features. The control conditions include task text encoding and / or visual features. The task text encoding is used to characterize the task to be performed by the task execution device, and the visual features are used to characterize the current surrounding environment information of the task execution device.

[0192] It should be understood that the operation and function of the acquisition module 910, prediction module 920, determination module 930, and training module 940 in the above embodiments can be referred to the above. Figure 6 The description of the model training method provided in the embodiments will not be repeated here to avoid repetition.

[0193] Figure 10 The diagram shown is a schematic representation of the structure of a model training apparatus provided in an exemplary embodiment of this application. Figure 10 As shown, the model training device 1000 includes: an acquisition module 1010, a first determination module 1020, a second determination module 1030, and a training module 1040.

[0194] The acquisition module 1010 is used to acquire second training samples, wherein the second training samples include second sample noise action features and second sample noise visual features. The first determination module 1020 is used to input the second training samples into the correlation information discriminant network sub-model to obtain second correlation information, wherein the second correlation information is used to characterize the degree of influence between the second sample noise visual features and the second sample noise action features. The second determination module 1030 is used to determine a second loss value based on the second correlation information and the target correlation information. The training module 1040 is used to train the correlation information discriminant network sub-model based on the second loss value to obtain the trained correlation information discriminant network sub-model.

[0195] This application provides a model training apparatus. By inputting the noise action features and noise visual features of a second sample into a correlation information discrimination network sub-model, predicted second correlation information is obtained. Based on the difference between the predicted second correlation information and the target correlation information, the trained correlation information discrimination network sub-model is obtained. The correlation information discrimination network sub-model obtained by this model training method has the ability to determine correlation information based on noise action features and noise visual features. Because the correlation information determined by the correlation information discrimination network sub-model is relatively high, the noise prediction sub-model can use information from different modalities to denoise based on the more accurate correlation information. This can, to a certain extent, ensure the prediction accuracy of action features.

[0196] According to an embodiment of this application, the second determining module 1030 is further configured to: input the second training sample and the first information into the trained noise prediction sub-model to obtain a predicted first feature, wherein the predicted first feature includes a predicted first action noise feature and a predicted first visual noise feature; determine a third loss value based on the difference between the predicted first action noise feature and the second target action noise feature, and the difference between the predicted first visual noise feature and the second target visual noise feature; input the second training sample and the second information into the trained noise prediction sub-model to obtain a predicted second feature, wherein the predicted second feature includes a predicted second action noise feature and a predicted second visual noise feature; determine a fourth loss value based on the difference between the predicted second action noise feature and the second target action noise feature, and the difference between the predicted second visual noise feature and the second target visual noise feature; determine the target correlation information as the first information when the third loss value is less than the fourth loss value; and determine the target correlation information as the second information when the third loss value is greater than the fourth loss value.

[0197] According to an embodiment of this application, the first determining module 1020 is used to input control conditions, first correlation information, first sample noise action features and first sample noise visual features into the noise prediction sub-model to obtain predicted action noise features and predicted visual noise features. The control conditions include task text encoding and / or visual features. The task text encoding is used to characterize the task to be performed by the task execution device, and the visual features are used to characterize the current surrounding environment information of the task execution device.

[0198] It should be understood that the operation and function of the acquisition module 1010, the first determination module 1020, the second determination module 1030, and the training module 1040 in the above embodiments can be referred to the above. Figure 7 The description of the model training method provided in the embodiments will not be repeated here to avoid repetition.

[0199] This application also provides a task execution device, which includes a control module for executing the motion feature prediction method provided in any of the above embodiments.

[0200] In some embodiments, the task execution device may be a robot, such as a dual-arm robot, or a vehicle, or a drone, or other device that can be used to perform a specific task.

[0201] The operation and functions of the task execution device provided in this application embodiment can be referred to the above. Figure 2 or Figure 5 The description of the action feature prediction method provided in the embodiments will not be repeated here to avoid repetition.

[0202] Figure 11 The diagram shown is a block diagram of an electronic device 1100 for performing an action feature prediction method or a model training method according to an exemplary embodiment of this application. Specifically, the electronic device 1100 may be a server, a computer, a task execution device, a control device for the task execution device, a server interacting with the task execution device, or other devices.

[0203] Reference Figure 11 The electronic device 1100 includes a processing component 1110, which further includes one or more processors, and memory resources represented by memory 1120 for storing instructions executable by the processing component 1110, such as application programs. The application programs stored in memory 1120 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1110 is configured to execute instructions to perform the aforementioned action feature prediction method or model training method.

[0204] Electronic device 1100 may also include a power supply component configured to perform power management of electronic device 1100, a wired or wireless network interface configured to connect electronic device 1100 to a network, and an input / output (I / O) interface. Electronic device 1100 can be operated based on an operating system stored in memory 1120, such as Windows Server. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0205] This application embodiment also provides a non-transitory computer-readable storage medium, which, when the instructions in the storage medium are executed by the processor of the aforementioned electronic device 1100, enables the electronic device 1100 to execute an action feature prediction method or a model training method.

[0206] This application also provides a computer program product, which includes a computer program. When the computer program is executed by the processor of a computer device, it enables the computer device to execute the motion feature prediction method or model training method provided in any of the above embodiments.

[0207] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0208] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0209] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0210] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0211] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0212] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0213] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program verification codes, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0214] It should be noted that in the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0215] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0216] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications or equivalent substitutions made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for predicting action features, characterized in that, include: During the denoising process, the first noise visual feature and the first noise motion feature at the current time step are obtained. The first noise visual feature is used to predict the surrounding environment information of the task execution device at future time, and the first noise motion feature is used to predict the motion information of the task execution device at future time. Determine the correlation information between the first noise visual feature and the first noise motion feature, wherein the correlation information is used to characterize the degree of influence between the first noise visual feature and the first noise motion feature; When the correlation information is the first information, bimodal interactive attention calculation is performed on the first noise visual feature and the first noise action feature to obtain the second noise visual feature and the second noise action feature. Based on the second noise visual feature and the second noise action feature, the predicted action feature is obtained. When the correlation information is the second information, self-attention calculation is performed on the first noise action feature to obtain the third noise action feature. Based on the third noise action feature, the predicted action feature is obtained. The influence between the first noise visual feature represented by the second information and the first noise action feature is lower than the influence between the first noise visual feature represented by the first information and the first noise action feature. The predicted action feature is used to control the action of the task execution device.

2. The motion feature prediction method according to claim 1, characterized in that, The denoising process includes T time steps, where the current time step is the t-th time step. Determining the correlation information between the first noise visual feature and the first noise motion feature includes: When the correlation information corresponding to the (t-1)th time step is the first information, the correlation information corresponding to the tth time step is determined based on the first noise visual feature and the first noise action feature at the tth time step, where t is an integer greater than 1; If the correlation information corresponding to the (t-1)th time step is the second information, then the correlation information corresponding to the tth time step is determined to be the second information.

3. The motion feature prediction method according to claim 2, characterized in that, The determination of the correlation information corresponding to the t-th time step based on the first noise visual features and the first noise motion features at the t-th time step includes: The first noise visual feature and the first noise action feature at time step t are input into the correlation information discrimination network sub-model to obtain the correlation information corresponding to time step t, which is output by the correlation information discrimination network sub-model. The step of performing bimodal interactive attention calculation on the first noise visual feature and the first noise action feature to obtain the second noise visual feature and the second noise action feature includes: The correlation information, the first noise visual feature, and the first noise action feature are input into the noise prediction sub-model to obtain the predicted visual noise feature and the predicted action noise feature output by the noise prediction sub-model. The first noise visual feature is denoised based on the predicted visual noise feature to obtain the second noise visual feature, and the first noise action feature is denoised based on the predicted action noise feature to obtain the second noise action feature, wherein the correlation information discrimination network sub-model and the noise prediction sub-model are constructed based on the diffusion model.

4. The motion feature prediction method according to claim 1, characterized in that, The denoising process includes T time steps, where the current time step is the t-th time step. The step of obtaining the predicted action features based on the second noise visual features and the second noise action features includes: The second noise visual feature and the second noise action feature are updated with the first noise visual feature and the first noise action feature at time step t+1, respectively. Based on the correlation information corresponding to time step t+1, the predicted action feature is obtained. The step of obtaining the predicted action features based on the third noise action features includes: The third noise action feature is updated to the first noise action feature at time step t+1, the correlation information corresponding to time step t+1 is determined as the second information, and the predicted action feature is obtained based on the third noise action feature corresponding to time step t+1, wherein the third noise action feature corresponding to time step T is used as the predicted action feature.

5. The motion feature prediction method according to claim 2, characterized in that, The determination of the correlation information corresponding to the t-th time step based on the first noise visual features and the first noise motion features at the t-th time step includes: Based on the control conditions, the first noise visual features and the first noise motion features at the t-th time step, the correlation information corresponding to the t-th time step is determined, wherein the control conditions include task text encoding and / or visual features, the task text encoding is used to characterize the task to be performed by the task execution device, and the visual features are used to characterize the current surrounding environment information of the task execution device.

6. The motion feature prediction method according to any one of claims 1 to 5, characterized in that, The step of performing bimodal interaction attention calculation on the first noise visual features and the first noise action features based on the correlation information, or performing self-attention calculation on the first noise action features to obtain predicted action features, includes: Based on the control conditions and the correlation information, bimodal interactive attention calculation is performed on the first noise visual features and the first noise action features, or self-attention calculation is performed on the first noise action features to obtain the predicted action features. The control conditions include task text encoding and / or visual features. The task text encoding is used to characterize the task to be performed by the task execution device, and the visual features are used to characterize the current surrounding environment information of the task execution device.

7. A model training method, characterized in that, include: Obtain a first training sample, wherein the first training sample includes a first target action noise feature, a first target visual noise feature, a first sample noise action feature, and a first sample noise visual feature; The first correlation information, the first sample noise action feature, and the first sample noise visual feature are input into the noise prediction sub-model to obtain the predicted action noise feature and the predicted visual noise feature. The first correlation information is used to characterize the degree of influence between the first sample noise action feature and the first sample noise visual feature. A first loss value is determined based on the difference between the predicted motion noise features and the first target motion noise features, and the difference between the predicted visual noise features and the first target visual noise features. The noise prediction sub-model is trained based on the first loss value to obtain the trained noise prediction sub-model. Wherein, when the denoising time step corresponding to the first sample noise action feature and the first sample noise visual feature is greater than the time step threshold, the first correlation information is the first information; when the denoising time step corresponding to the first sample noise action feature and the first sample noise visual feature is less than or equal to the time step threshold, the first correlation information is the second information or the first information, wherein the degree of influence between the first sample noise action feature and the first sample noise visual feature represented by the second information is lower than the degree of influence between the first sample noise action feature and the first sample noise visual feature represented by the first information.

8. The model training method according to claim 7, characterized in that, Also includes: Obtain a second training sample, wherein the second training sample includes second sample noise action features and second sample noise visual features; The second training sample is input into the correlation information discriminant network sub-model to obtain the second correlation information, wherein the second correlation information is used to characterize the degree of influence between the second sample noise visual features and the second sample noise action features; The second loss value is determined based on the second correlation information and the target correlation information; The correlation information discrimination network sub-model is trained based on the second loss value to obtain the trained correlation information discrimination network sub-model.

9. The model training method according to claim 8, characterized in that, Also includes: The second training sample and the first information are input into the trained noise prediction sub-model to obtain the predicted first feature, wherein the predicted first feature includes the predicted first action noise feature and the predicted first visual noise feature; A third loss value is determined based on the difference between the predicted first action noise feature and the second target action noise feature, and the difference between the predicted first visual noise feature and the second target visual noise feature; The second training sample and the second information are input into the trained noise prediction sub-model to obtain the predicted second feature, wherein the predicted second feature includes the predicted second action noise feature and the predicted second visual noise feature; A fourth loss value is determined based on the difference between the predicted second action noise feature and the second target action noise feature, and the difference between the predicted second visual noise feature and the second target visual noise feature; If the third loss value is less than the fourth loss value, the target correlation information is determined to be the first information; If the third loss value is greater than the fourth loss value, the target correlation information is determined to be the second information.

10. The model training method according to any one of claims 7 to 9, characterized in that, The step of inputting the first correlation information, the first sample noise action features, and the first sample noise visual features into the noise prediction sub-model to obtain the predicted action noise features and the predicted visual noise features includes: The control conditions, the first correlation information, the first sample noise action features, and the first sample noise visual features are input into the noise prediction sub-model to obtain the predicted action noise features and the predicted visual noise features. The control conditions include task text encoding and / or visual features. The task text encoding is used to characterize the task to be performed by the task execution device, and the visual features are used to characterize the current surrounding environment information of the task execution device.

11. A model training method, characterized in that, include: Obtain a second training sample, wherein the second training sample includes second sample noise action features and second sample noise visual features; The second training sample is input into the correlation information discriminant network sub-model to obtain the second correlation information, wherein the second correlation information is used to characterize the degree of influence between the second sample noise visual features and the second sample noise action features; The second loss value is determined based on the second correlation information and the target correlation information; The correlation information discrimination network sub-model is trained based on the second loss value to obtain the trained correlation information discrimination network sub-model. Wherein, when the denoising time step corresponding to the second sample noise action feature and the second sample noise visual feature is greater than the time step threshold, the second correlation information is the first information; when the denoising time step corresponding to the second sample noise action feature and the second sample noise visual feature is less than or equal to the time step threshold, the second correlation information is the second information or the first information, wherein the degree of influence between the second sample noise action feature and the second sample noise visual feature represented by the second information is lower than the degree of influence between the second sample noise action feature and the second sample noise visual feature represented by the first information.

12. A task execution device, characterized in that, It includes a control module, which is used to execute the motion feature prediction method according to any one of claims 1 to 6.

13. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions. The processor is used to execute the action feature prediction method according to any one of claims 1 to 6 or the model training method according to any one of claims 7 to 11.

14. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the action feature prediction method of any one of claims 1 to 6 or the model training method of any one of claims 7 to 11.

15. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by the processor of a computer device, enables the computer device to perform the action feature prediction method of any one of claims 1 to 6 or the model training method of any one of claims 7 to 11.

Images