Processing method and computer readable storage medium for intelligent agent motion control

Abstract

This invention discloses a processing method and a computer-readable storage medium for intelligent agent motion control. The processing method includes: acquiring control commands, visual observation data collected at a first time step, and a first proprioceptive state of the intelligent agent, wherein the visual observation data includes a first image observed at the first time step; inputting the control commands and visual observation data into a prediction subsystem to obtain a first predicted intention feature of the predicted image to be observed at a second time step, wherein the second time step is the Hth time step after the first time step; inputting the first predicted intention feature and the first proprioceptive state into an action subsystem to obtain control information for controlling the intelligent agent, wherein the control information is used to drive the joint movements of the intelligent agent to realize the change in motion state between the first and second time steps indicated by the control commands. This processing method avoids the loss of generalization ability caused by directly fitting motion noise.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligent agent technology, and in particular to a processing method and a computer-readable storage medium for motion control of intelligent agents. Background Technology

[0002] In the field of general embodied intelligent agents, Vision-Language-Action (VLA) models have become the mainstream paradigm for mapping high-level reasoning to motion control. Different VLA architectures exist within this field, such as end-to-end VLAs, which directly use action labels to provide end-to-end supervision of the visual-language model. However, these models suffer from training instability. Direct action gradient backpropagation can lead to potential spatial collapse during visual-language model training, causing the model to learn gradient shortcuts—ignoring high-level semantic reasoning and instead fitting low-level visual noise—significantly reducing the model's generalization ability. Summary of the Invention

[0003] The present invention provides a processing method and a computer-readable storage medium for intelligent agent motion control to solve at least one of the aforementioned technical problems.

[0004] This invention provides a processing method for motion control of an intelligent agent. The processing method includes: acquiring a control command, visual observation data collected at a first time step, and a first proprioceptive state of the intelligent agent, wherein the visual observation data includes a first image observed at the first time step; inputting the control command and the visual observation data into a prediction subsystem to obtain a first predicted intent feature of a predicted image observed at a second time step, wherein the second time step is the H-th time step after the first time step, and H is a positive integer; inputting the first predicted intent feature and the first proprioceptive state into an action subsystem to obtain control information for controlling the intelligent agent, wherein the control information is used to drive joint movements of the intelligent agent to realize the change in motion state between the first time step and the second time step indicated by the control command.

[0005] In the above processing method, the prediction subsystem does not directly output action signals, but predicts the future characteristics of the agent. The action subsystem can be used as an inverse dynamic model to deduce actions based on the future characteristics of the agent, thereby constructing a structured bottleneck of "intention-action". Compared with the traditional end-to-end VLA model, it can avoid the loss of generalization ability caused by directly fitting action noise, which is conducive to improving the generalization of the overall system composed of the prediction subsystem and the action subsystem when used for agent action control.

[0006] In one optional technical solution of the present invention, the control information includes at least one of the following: a first action block, which corresponds to the motion state of the agent at the second time step; and at least two second action blocks, which respectively correspond to the motion state of the agent at one of the H time steps following the first time step. This improves the applicability of agent action prediction.

[0007] In one optional technical solution of the present invention, the step of inputting the control command and the visual observation data into the prediction subsystem to obtain a first predictive intent feature of the predicted image to be observed in the second time step includes: extracting the current visual features corresponding to the first image in the visual observation data through a visual encoder in the prediction subsystem; inputting the control command and the current visual features into a visual language model in the prediction subsystem, and outputting the first predictive intent feature. In this way, the prediction effect of the agent's motion state in the second time step can be achieved.

[0008] In one optional technical solution of the present invention, the step of inputting the first predicted intent feature and the first ontology perception state into the action subsystem to obtain control information for controlling the agent includes: fusing the current visual features with the first predicted intent feature through an attention mechanism in the action subsystem to obtain a first fused feature; combining the first fused feature with the first ontology perception state and inputting it into the action generation model in the action subsystem to generate the control information. In this way, the effect of deriving the agent's actions based on the prediction results can be achieved.

[0009] In one optional technical solution of the present invention, the first body perception state includes the joint angle of the intelligent agent; The first action block or the second action block includes the change in the joint angle, which characterizes the joint motion parameters required for the agent to change from its current motion state to the motion state corresponding to the first action block or the second action block. This allows for a simple way to drive the agent to perform motion control.

[0010] In one optional technical solution of the present invention, the processing method includes: independently pre-training the prediction subsystem and the action subsystem based on the acquired training samples, wherein the training samples include a first sample image and a second sample image, the first sample image being a third image observed at a third time step, and the second sample image being a fourth image observed at a fourth time step, the fourth time step being the h-th time step after the third time step, where h is a positive integer; after the independent pre-training is completed, the prediction subsystem and the action subsystem are co-trained. This facilitates decoupling between the prediction subsystem and the action subsystem.

[0011] In an optional technical solution of the present invention, the step of independently pre-training the prediction subsystem and the action subsystem based on the acquired training samples includes: inputting a first visual feature and a linguistic feature into the prediction subsystem to obtain a second predictive intent feature that is expected to be observed in the second sample image at a fourth time step, wherein the first visual feature is obtained by encoding the first sample image, and the linguistic feature corresponds to the training instruction issued in the training scenario; adjusting the parameters of the prediction subsystem according to the second visual feature and the second predictive intent feature, wherein the second visual feature is obtained by encoding the second sample image. This improves the accuracy and stability of the prediction subsystem in predicting actions.

[0012] In an optional technical solution of the present invention, inputting the first visual feature and linguistic feature into the prediction subsystem includes: inputting at least one learnable first query term, the first visual feature, and the linguistic feature into the prediction subsystem to obtain the second predicted intent feature, wherein the first query term is used by the prediction subsystem to query semantics and vision. This enriches the common sense used in the prediction subsystem to represent semantics and vision, thereby improving the accuracy of intent prediction processing.

[0013] In an optional technical solution of the present invention, adjusting the parameters of the prediction subsystem based on the second visual feature and the second prediction intent feature includes: obtaining the mean square error between the second prediction intent feature and the second visual feature; and adjusting the parameters of the prediction subsystem if the mean square error is greater than a preset error. This prevents gradient interference during training.

[0014] In an optional technical solution of the present invention, the step of independently pre-training the prediction subsystem and the action subsystem based on the acquired training samples includes: fusing a first visual feature and a second visual feature to obtain a second fused feature, wherein the first visual feature and the second visual feature are obtained by encoding the first sample image and the second sample image, respectively, and the feature spaces of the first visual feature and the second visual feature are consistent; inputting the second ontology perception state of the agent and the second fused feature into the action subsystem to obtain the control information, wherein the second ontology perception state is the ontology perception state of the agent in the training scenario. This avoids processing high-dimensional pixel noise.

[0015] In one optional technical solution of the present invention, the step of fusing the first visual feature and the second visual feature to obtain the second fused feature includes: performing self-attention processing on the first visual feature and the second visual feature; and performing cross-attention processing on the self-attention processed first visual feature and second visual feature to obtain the second fused feature. This increases the correlation between the first visual feature and the second visual feature.

[0016] In an optional technical solution of the present invention, the step of independently pre-training the prediction subsystem and the action subsystem based on the acquired training samples further includes: inputting the second ontology perception state into the regression model of the action subsystem to obtain vector data, wherein the vector data is used to represent the vector corresponding to the second ontology perception state in the feature space; and inputting the second ontology perception state of the agent and the second fused feature into the action subsystem, which includes: inputting the vector data, the second ontology perception state, and the second fused feature into the action generation model of the action subsystem. In this way, the joint angles of the agent in the training scenario can be mapped to the feature space, facilitating processing in conjunction with other inputs.

[0017] In one optional technical solution of the present invention, inputting the agent's second ontology perception state and the second fusion feature into the action subsystem includes: inputting at least one learnable second query term, the second ontology perception state, and the second fusion feature into the action generation model of the action subsystem, wherein the second query term is used by the action subsystem to query semantics and vision. This enriches the common sense used to represent semantics and vision in the action subsystem, thereby improving the accuracy of action prediction processing.

[0018] In one optional technical solution of the present invention, the co-training of the prediction subsystem and the action subsystem includes: concatenating the prediction subsystem and the action subsystem; and jointly training the concatenated prediction subsystem and action subsystem using the first sample image, wherein the loss function of the joint training includes action prediction loss and intention prediction loss. This prevents feature space collapse and gradient shortcuts from occurring during system training.

[0019] The present invention provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the processing method described in any of the above optional technical solutions.

[0020] In the aforementioned computer equipment, the prediction subsystem does not directly output action signals, but instead predicts the future characteristics of the agent. The action subsystem can be used as an inverse dynamics model to deduce actions based on the future characteristics of the agent, thereby constructing a structured bottleneck of "intention-action". Compared with the traditional end-to-end VLA model, it can avoid the loss of generalization ability caused by directly fitting action noise, which is conducive to improving the generalization of the overall system composed of the prediction subsystem and the action subsystem when used for agent action control.

[0021] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart illustrating the processing method of the optional technical solution of the present invention; Figure 2 This is a schematic diagram of the processing system of the optional technical solution of the present invention; Figure 3 This is a schematic diagram of the first image of an optional technical solution of the present invention; Figure 4 This is a schematic diagram of the predicted image of an optional technical solution of the present invention; Figure 5 This is another flowchart illustrating the processing method of the optional technical solution of the present invention; Figure 6 This is a schematic diagram of the prediction subsystem and action subsystem of the optional technical solution of the present invention in a training scenario; Figure 7 This is another flowchart illustrating the processing method of the optional technical solution of the present invention; Figure 8This is another schematic diagram of the processing method of the optional technical solution of the present invention; Figure 9 This is another schematic diagram of the processing method of the optional technical solution of the present invention; Figure 10 This is a schematic diagram of a computer device according to an optional technical solution of the present invention.

[0023] Explanation of key component symbols: Processing system 100; Prediction subsystem 110, visual language model 111, first regression model 112; Action subsystem 120, self-attention module 121, cross-attention module 122, action generation model 123, second regression model 124; First encoder 131, second encoder 132, preset visual encoder 133; Intelligent agent 200, robot 210, left hand 211, right hand 212; Water cup 310, water bottle 320; First sample image 410, second sample image 420; Computer equipment 500, memory 510, processor 520. Detailed Implementation

[0024] In the description of this invention, some of the disclosed content has been shown accordingly in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The following description with reference to the accompanying drawings is exemplary and is only used to explain the invention, and should not be construed as limiting the invention.

[0025] In the description of this invention, many different contents or examples are disclosed to implement different structures of the invention. To simplify the disclosure of this invention, the components and arrangements of specific examples are described below. Of course, these are merely examples and are not intended to limit the invention.

[0026] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0027] In the field of general embodied intelligent agents, the Vision-Language-Action (VLA) model has become the mainstream paradigm for mapping high-level reasoning to motion control. However, current VLA architectures face a serious "trilemma" in terms of generalization, stability, and accuracy. Currently, related methods mainly fall into three technical routes, each with the following significant bottlenecks: 1. Hierarchical models: These models utilize a Vision-Language Model (VLM) for abstract planning and output text, which is then executed by the underlying controller. Their drawbacks include: reliance on autoregressive text generation leading to extremely high inference latency; and the inability of text, as a low-bandwidth medium, to capture the fine-grained physical details required for complex operations (lack of precision). 2. End-to-End VLAs: These models directly supervise the VLM (Visual Model) end-to-end using action labels. Their drawback is severe training instability. Direct action gradient backpropagation can easily lead to spatial collapse of the pre-trained VLM, causing the model to learn "gradient shortcuts"—ignoring high-level semantic reasoning and instead fitting low-level visual noise, thus significantly reducing the model's generalization ability. 3. World Action Models (VLMs): These models plan by predicting the future. Their drawbacks include: predictions based on generative backbones (such as video diffusion models) incur high computational latency, and they lack the rich semantic common sense inherent in pre-trained VLMs. Furthermore, existing VLA-based world action models often treat "future prediction" as merely a peripheral auxiliary task, failing to architecturally force action generation to depend on the expected future, leading to a disconnect between intent and action.

[0028] Building upon the above, this invention aims to address the aforementioned "trilemma" by proposing a framework called DIAL (Decoupling Intent and Action via Latent World Modeling). The technical problem this invention solves is: how to structurally decouple the model's high-level "future imagination (intent)" from its low-level "precise execution (action)" while maintaining end-to-end differentiable optimization, thereby achieving efficient, high-frequency, and high-precision robot motion control without compromising the semantic generalization ability of large-scale models.

[0029] Please refer to Figure 1 A processing method for motion control of an intelligent agent 200 according to the present invention may include: 03: Acquire control commands, visual observation data collected at the first time step, and the first ontological perception state of the agent 200. The visual observation data includes the first image observed at the first time step. 04: Input the control command and visual observation data into the prediction subsystem 110 to obtain the first prediction intention feature of the predicted image expected to be observed at the second time step. The second time step is the Hth time step after the first time step, where H is a positive integer. 05: Input the first predicted intent feature and the first ontological perception state into the action subsystem 120 to obtain control information for controlling the agent 200. The control information is used to drive the joint movement of the agent 200 to realize the change in motion state between the first time step and the second time step indicated by the control command.

[0030] The processing method of the present invention can be implemented by the processing system 100 of the present invention. Specifically, please refer to... Figure 2 A processing system 100 for motion control of an intelligent agent 200 according to the present invention may include a prediction subsystem 110 and an action subsystem 120. The prediction subsystem 110 may be used to: acquire control commands and visual observation data collected at a first time step, the visual observation data including a first image observed at the first time step; and use the control commands and visual observation data as input to obtain a first predicted intention feature of a predicted image to be observed at a second time step, where the second time step is the H-th time step after the first time step, and H is a positive integer. The action subsystem 120 may be used to: acquire a first proprioceptive state of the intelligent agent 200; and use the first predicted intention feature and the first proprioceptive state as input to obtain control information for controlling the intelligent agent 200, the control information being used to drive joint movements of the intelligent agent 200 to realize the change in motion state between the first and second time steps indicated by the control commands.

[0031] In the above processing method and processing system 100, the prediction subsystem 110 does not directly output action signals, but predicts the future characteristics of the agent 200. The action subsystem 120 can be used as an inverse dynamics model to deduce actions based on the future characteristics of the agent 200, thereby constructing a structured bottleneck of "intention-action". Compared with the traditional end-to-end VLA model, it can avoid the loss of generalization ability caused by directly fitting action noise, which is conducive to improving the generalization of the overall system composed of prediction subsystem 110 and action subsystem 120 when used for action control of agent 200.

[0032] Specifically, please combine Figure 3 In such an application scenario, the intelligent agent 200 may include a robot 210. The robot 210 may have a left hand 211 and a right hand 212. Figure 3In the diagram, robot 210 holds an empty water cup 310 in its left hand 211 and a water bottle 320 filled with water in its right hand 212. A control command can be issued to control robot 210 to pour water. The control command can be issued by the user or through a pre-set program.

[0033] Within the first time step, control commands can be received by the processing system 100. In the application scenario, images can be continuously acquired, thus allowing the first image to be observed within the first time step. The first image can be further processed to obtain visual observation data. The images can be acquired by vision sensors mounted on the robot 210, or by cameras around or facing the robot 210 capturing images of the robot 210.

[0034] Accordingly, in the application scenario, the joint angles of robot 210 can be continuously collected, and these joint angles can be packaged or integrated to serve as the first body perception state. The first body perception state can be obtained through motion sensors installed on robot 210.

[0035] After acquiring control commands and visual observation data, the prediction subsystem 110 can process the control commands into corresponding text information and the visual observation data into corresponding visual information as input. This allows it to combine the control commands and visual observation data to predict the robot 210's intention to perform the water-pouring action at the first time step. Specifically, it can predict what operation the robot 210 needs to perform next to achieve the water-pouring action, or, in other words, predict the robot 210's motion state at a future time step (such as the second time step). Figure 4 (As shown). In Figure 4 In the middle, the robot 210 is in the action of controlling its right hand 212 to operate the water bottle 320 to pour water into the water cup 310 in its left hand 211. Figure 4 It can be considered as predicting the motion state of robot 210 at the second time step, or in other words, Figure 4 It can be used to predict images.

[0036] After obtaining the first predicted intent feature, the first predicted intent feature can be output to the motion subsystem 120. Combined with the first proprioceptive state of the robot 210, the motion subsystem 120 can process the robot 210 with the joint angles at the first time step as the initial state and the predicted motion state of the robot 210 at the second time step as the target state to obtain control information. The control information can correspond to the robot 210 controlling its own joints (including the joints of the robot 210's head, waist, arms, wrists, fingers, etc.) to change the angle changes of the limb parts. For example, controlling the right hand 212 to raise and the left hand 211 to fall, and then controlling the right hand 212 to twist so that the opening of the water bottle 320 moves closer to the left hand 211, and finally so that the opening of the water bottle 320 faces the opening of the water cup 310, so that the water bottle 320 pours water into the water cup 310.

[0037] Based on the above, the prediction subsystem 110 predicts the robot 210's actions for the unexpected second time step (H time steps after the first time step), i.e., predicts the future features of the agent 200. The action subsystem 120 then derives the agent 200's actions based on the future features predicted by the prediction subsystem 110. The derived actions can be used to control the agent 200 to perform corresponding actions, so that the agent 200 can be in the expected motion state in the predicted image at the second time step. The prediction subsystem 110 and the action subsystem 120 constitute the structured bottleneck of "intent-action". Compared with related technologies (such as end-to-end VLA models), this avoids the loss of generalization ability caused by directly fitting action noise, which is beneficial to improving the generalization of the overall system composed of the prediction subsystem 110 and the action subsystem 120 when used for the action control of the agent 200.

[0038] In addition, the intelligent agent 200 may also include other intelligent devices (such as robot dogs) or carriers equipped with intelligent devices (such as unmanned vehicles).

[0039] In some cases, the control information may include at least one of the following: a first action block, which corresponds to the motion state of the agent 200 at a second time step; at least two second action blocks, which respectively correspond to the motion state of the agent 200 at one of the H time steps following the first time step.

[0040] This improves the applicability of agent 200 action prediction.

[0041] Specifically, depending on the specific needs in different application scenarios, if the second time step is the next time step after the first time step (H is 1), that is, only the motion state under the second time step needs to be known, then only the relevant information of the second time step (i.e., the first action block) can be output; if the second time step and the first time step are separated by several time steps (H is greater than 1), that is, the motion state under H time steps after the first time step needs to be known, then the relevant information of the H time steps after the first time step (i.e., at least two second action blocks) can be output.

[0042] In some cases, step 05 (inputting the first predicted intent feature and the first prophetic perception state to the action subsystem 120 to obtain control information for controlling the agent 200) may include: Obtain the current visual features corresponding to the first image; Calculate the feature residual between the current visual features and the first predicted potential intent features; The motion subsystem 120 generates the change in joint angle based on the feature residual and the first body perception state, and generates control information based on the change in joint angle.

[0043] The processing method of the present invention can be implemented by the processing system 100 of the present invention. Specifically, please refer to... Figure 2 The action subsystem 120 can be used to acquire the current visual features corresponding to the first image; calculate the feature residual between the current visual features and the first predicted potential intent features; generate the change in joint angle based on the feature residual and the first ontology perception state through the action subsystem 120, and generate control information based on the change in joint angle.

[0044] This helps improve the accuracy of motion control of the agent 200 by predicting intentions and inferring actions.

[0045] Specifically, the action subsystem 120 can use the current visual features as input and the first predicted latent intention features as output to calculate and determine the feature residual between the two. The feature residual can represent the direction and / or magnitude of change in the latent space from the current state (corresponding to the first image observed at the first time step) to the future state (corresponding to the predicted image expected to be observed at the second time step), determining the difference between the agent 200's current state and future state. Based on this difference, it determines the state or posture the agent 200 should be in at the future second time step, and how motion control should be performed between the first and second time steps, so that the agent 200's motion state at the first and second time steps maintains the difference obtained through the predicted intention. Furthermore, the output control information is adjusted based on the obtained difference, so that the control information enables the agent 200 to perform motion control according to this difference.

[0046] In some cases, the processing methods may include: Calculate the feature residual between the current visual features corresponding to the first image and the first predicted potential intent features; The feature residuals and the first body perception state are input to the motion subsystem 120 to generate the change in joint angles, and control information is generated based on the change in joint angles.

[0047] The processing method of the present invention can be implemented by the processing system 100 of the present invention. Specifically, please refer to... Figure 2 The prediction subsystem 110 can be used to calculate the feature residual between the current visual features corresponding to the first image and the first predicted potential intention features; input the feature residual and the first ontology perception state to the action subsystem 120 to generate the change in joint angle, and generate control information based on the change in joint angle.

[0048] This helps improve the accuracy of motion control of the agent 200 by predicting intentions and inferring actions.

[0049] Specifically, the prediction subsystem 110 can use the current visual features as input and the first predicted latent intention features as output to calculate and determine the feature residual between the two. The feature residual can represent the direction and / or magnitude of change in the latent space from the current state (corresponding to the first image observed at the first time step) to the future state (corresponding to the predicted image expected to be observed at the second time step), determining the difference between the agent 200's current state and future state. Based on this difference, it determines the state or posture the agent 200 should be in at the future second time step, and how motion control should be performed between the first and second time steps, so that the agent 200's motion state at the first and second time steps maintains the difference obtained through the predicted intention. Furthermore, the output control information is adjusted based on the obtained difference, so that the control information enables the agent 200 to perform motion control according to this difference.

[0050] It should be noted that, unlike the aforementioned method of generating control information through feature residuals, the method here is to obtain feature residuals through prediction subsystem 110, and then output feature residuals to action subsystem 120. In this way, prediction subsystem 110 and action subsystem 120 can undertake different processing tasks in the scheme of generating control information through feature residuals, thereby reducing the pressure on the computing resources of action subsystem 120.

[0051] Please refer to Figure 5 In some cases, the processing methods may include: 01: The prediction subsystem 110 and the action subsystem 120 are independently pre-trained based on the obtained training samples. The training samples include a first sample image 410 and a second sample image 420. The first sample image 410 is the third image observed at the third time step, and the second sample image 420 is the fourth image observed at the fourth time step. The fourth time step is the h-th time step after the third time step, where h is a positive integer. 02: After independent pre-training is completed, the prediction subsystem 110 and the action subsystem 120 are trained together.

[0052] The processing method of the present invention can be implemented by the processing system 100 of the present invention. Specifically, please refer to... Figure 2 The prediction subsystem 110 can be used for independent pre-training based on the acquired training samples; after independent pre-training, it can be co-trained with the action subsystem 120.

[0053] This facilitates the decoupling between the prediction subsystem 110 and the action subsystem 120.

[0054] Specifically, please combine Figure 6 ,exist Figure 6 In the training scenario, the first sample image 410 can be used as the input of the prediction subsystem 110, and the second sample image 420 can be used as the label of the prediction subsystem 110 in the training scenario. This allows the prediction subsystem 110 to return and adjust parameters based on the error between the output and the label, so that the output of the prediction subsystem 110 can be as close as possible to the actual situation.

[0055] And, in Figure 6 In the training scenario, the second sample image 420 can be used as the input of the action subsystem 120, so that the action subsystem 120 can deduce the action based on the actual action state.

[0056] After completing independent pre-training, the prediction subsystem 110 and action subsystem 120 can be combined for collaborative training. Figure 6 During collaborative training, the action subsystem 120 can use the prediction results output by the prediction subsystem 110 as input until collaborative training is completed.

[0057] Based on the above, although the prediction subsystem 110 and the action subsystem 120 use the same samples in independent pre-training, their training processes are independent. This results in a structural decoupling effect between the prediction subsystem 110 and the action subsystem 120. The prediction subsystem 110 can predict actions without relying on the action subsystem 120, and the action subsystem 120 can deduce actions without relying on the prediction subsystem 110.

[0058] Please combine Figure 7 In some cases, step 01 (independent pre-training of the prediction subsystem 110 and the action subsystem 120 based on the obtained training samples) may include: 011: Input the first visual feature and the language feature into the prediction subsystem 110 to obtain the second prediction intention feature of the second sample image 420 expected to be observed at the fourth time step. The first visual feature is obtained by encoding the first sample image 410, and the language feature corresponds to the training instruction issued in the training scenario. 012: Adjust the parameters of the prediction subsystem 110 based on the second visual feature and the second prediction intention feature. The second visual feature is obtained by encoding the second sample image 420.

[0059] The processing method of the present invention can be implemented by the processing system 100 of the present invention. Specifically, please refer to... Figure 2The prediction subsystem 110 can be used to input the first visual feature and the language feature into the prediction subsystem 110 to obtain the second prediction intention feature of the second sample image 420 expected to be observed at the fourth time step. The first visual feature is obtained by encoding the first sample image 410, and the language feature corresponds to the training instruction issued in the training scenario. The parameters of the prediction subsystem 110 are adjusted according to the second visual feature and the second prediction intention feature. The second visual feature is obtained by encoding the second sample image 420.

[0060] This helps improve the accuracy and stability of the prediction subsystem 110 when predicting actions.

[0061] Specifically, in Figure 6 In this system, the prediction subsystem 110 may include a visual language model 111. The prediction subsystem 110 takes first visual features and language features as input, and can input these features into the visual language model 111. The visual language model 111 interprets the image information contained in the first visual features and the text information contained in the language features to obtain the intention related to the action of the agent 200, i.e., the second predicted intention feature. The visual language model 111 may include a large-scale visual language model (LLM).

[0062] Furthermore, after the prediction subsystem 110 obtains the second prediction intention feature, it can use the second prediction intention feature as the output and the second visual feature as the label. Based on the error between the output and the label, it returns to adjust the parameters of the prediction subsystem 110 itself, so that the prediction subsystem 110 after parameter adjustment can output a result that is closer to the second visual feature.

[0063] Based on the above, the prediction subsystem 110 can not only focus on the accuracy of the underlying action prediction, but also needs to pay attention to visual features and imagine future world changes, which is conducive to improving the robustness (stability) of the entire system.

[0064] In some cases, step 04 (inputting control commands and visual observation data into the prediction subsystem 110 to obtain a first predictive intent feature of the predicted image to be observed at the second time step) may include: The visual encoder in the prediction subsystem 110 extracts the current visual features corresponding to the first image in the visual observation data; The control command and the current visual features are input into the visual language model 111 in the prediction subsystem 110, and the first predicted intent feature is output.

[0065] The processing method of the present invention can be implemented by the processing system 100 of the present invention. Specifically, please refer to... Figure 2The prediction subsystem 110 can be used to extract the current visual features corresponding to the first image in the visual observation data through the visual encoder in the prediction subsystem 110; input the control command and the current visual features into the visual language model 111 in the prediction subsystem 110, and output the first prediction intention feature.

[0066] In this way, the motion state of agent 200 at the second time step can be predicted.

[0067] Specifically, in the application scenario, after the prediction subsystem 110 receives the first image, it can extract the current visual features from the first image through the visual encoder, and then input the current visual features and control commands into the visual language model 111. Since the visual language model 111 can be pre-trained to have high prediction accuracy, the first prediction intention feature it outputs is very close to the visual features corresponding to the prediction image that can be actually observed in the second time step. Thus, it is possible to predict the intention of the operation that the agent 200 is expected to perform in the second time step and determine the motion state of the agent 200 in the second time step.

[0068] In some cases, step 011 (inputting the first visual features and linguistic features into the prediction subsystem 110) may include: At least one learnable first query term, a first visual feature, and a linguistic feature are input into the prediction subsystem 110 to obtain a second prediction intent feature. The first query term is used by the prediction subsystem 110 to query semantics and vision.

[0069] The processing method of the present invention can be implemented by the processing system 100 of the present invention. Specifically, please refer to... Figure 2 The prediction subsystem 110 can be used to input at least one learnable first query term, a first visual feature and a linguistic feature into the prediction subsystem 110 to obtain a second prediction intent feature. The first query term is used for the prediction subsystem 110 to query semantics and vision.

[0070] This enriches the common sense used to represent semantics and vision in the prediction subsystem 110, thereby improving the accuracy of intent prediction processing.

[0071] Specifically, the first query term may include randomly initialized first continuous features. Multiple first continuous features may have continuously varying coordinate positions or continuously varying vector angles within the feature space.

[0072] Please combine Figure 6Specifically, the visual language model 111 can have a high-dimensional feature space, containing a large number of high-dimensional vectors. These vectors have both a starting point and a direction within the feature space. The visual language model 111's intention prediction processing of the first visual feature and the language feature can query word meanings that are close to the first visual feature and the language feature respectively in the high-dimensional feature space, thereby understanding the intention in the first visual feature and the language feature.

[0073] Based on the above, by inputting the first query term into the visual language model 111, a large number of terms that can continuously change in coordinate position are added to the feature space of the visual language model 111, and / or a large number of terms that can continuously change in vector direction are added, which helps to improve the semantic and visual common sense richness of the visual language model 111, and thus can more accurately understand the intent in the first visual features and language features.

[0074] In addition, it can be understood that for the visual language model 111, based on receiving the first visual features and language features as input, by adding the first query term, a look-ahead bridge can be formed. After using the visual language model 111 to process multimodal input, the output vector of the first query term can be directly trained as a visual feature that approximates a certain step (or multiple steps) of the future environment.

[0075] Furthermore, by predicting features rather than raw pixels or text, the visual language model 111 possesses the foresight of a world model while maintaining the reasoning speed of a reactive agent. This facilitates the combination of the "common sense" of the high-level visual language model 111 and the "flexibility" of the low-level action expert model to achieve complementary advantages.

[0076] In some cases, please combine Figure 6 ,exist Figure 6 In this process, the first sample image can be encoded using the first encoder 131, and the second sample image can be encoded using the second encoder 132. The first encoder 131 and the second encoder 132 can be obtained by multiplexing a preset visual encoder 133. In some cases, the first encoder 131 and the second encoder 132 can be the same visual encoder.

[0077] In this way, the same encoder can be used to extract visual features from the first sample image 410 and the second sample image 420, ensuring that the first visual features and the second visual features are in a unified manifold space, thus ensuring the stability of optimization during the training process.

[0078] Specifically, in Figure 6In the prediction subsystem 110, a preset visual encoder 133 may be included. In the training scenario, the preset visual encoder 133 can be reused to generate a first encoder 131 and a second encoder 132. This makes the first encoder 131 and the second encoder 132 have the same structure, so that the visual data obtained by visually encoding the image through the first encoder 131 and the second encoder 132 respectively have a consistent feature space. The underlying module directly compares feature differences in the manifold space without performing cross-modal alignment operations, and directly outputs high-frequency low-level action control commands, thereby improving the stability of training optimization.

[0079] In addition, the first encoder 131, the second encoder 132 and the preset visual encoder 133 may include a ViT encoder, a SigLIP, or a strong geometric prior visual encoder.

[0080] In some cases, step 012 (adjusting the parameters of the prediction subsystem 110 based on the second visual features and the second prediction intent features) may include: Obtain the mean squared error between the second predicted intent feature and the second visual feature; If the mean square error is greater than the preset error, the parameters of the prediction subsystem 110 are adjusted.

[0081] The processing method of the present invention can be implemented by the processing system 100 of the present invention. Specifically, please refer to... Figure 2 The prediction subsystem 110 can be used to obtain the mean square error between the second prediction intention feature and the second visual feature; if the mean square error is greater than the preset error, the parameters of the prediction subsystem 110 are adjusted.

[0082] This prevents gradient interference during training.

[0083] Specifically, in Figure 6 In this system, the prediction subsystem 110 may include a first regression model 112. After obtaining the second predicted intent feature, the first regression model 112 can perform regression processing on the second predicted intent feature based on the second visual feature, determine the difference between the second predicted intent feature and the second visual feature, and then evaluate its own model performance based on the difference result, thereby enabling the prediction subsystem 110 to adjust the parameters of its own visual language model 111 and / or the first regression model 112. The first regression model 112 may include an MLP (Multilayer Perceptron).

[0084] It is understandable that by checking whether the mean square error between the second predicted intent feature and the second visual feature is greater than a preset error, it is determined whether to adjust the parameters of the prediction subsystem 110. This achieves the effect of the prediction subsystem 110 learning MSE loss, so that the prediction subsystem 110 does not completely rely on the gradient update brought by the action prediction loss of the action subsystem 120, thus avoiding the collapse of the semantic representation space of the prediction subsystem 110 and preventing gradient interference during training.

[0085] Please refer to Figure 8 In some cases, step 01 (independent pre-training of the prediction subsystem 110 and the action subsystem 120 based on the obtained training samples) may include: 013: The first visual feature and the second visual feature are fused to obtain the second fused feature. The first visual feature and the second visual feature are obtained by encoding the first sample image 410 and the second sample image 420, respectively. The feature spaces of the first visual feature and the second visual feature are consistent. 014: Input the second ontology perception state and the second fusion feature of the agent 200 into the action subsystem 120 to obtain control information. The second ontology perception state is the ontology perception state of the agent 200 in the training scenario.

[0086] The processing method of the present invention can be implemented by the processing system 100 of the present invention. Specifically, please refer to... Figure 2 The action subsystem 120 can be used to fuse the first visual feature and the second visual feature to obtain the second fused feature. The first visual feature and the second visual feature are obtained by encoding the first sample image 410 and the second sample image 420, respectively. The feature spaces of the first visual feature and the second visual feature are consistent. The second ontology perception state of the agent 200 and the second fused feature are input to the action subsystem 120 to obtain control information. The second ontology perception state is the ontology perception state of the agent 200 in the training scenario.

[0087] In this way, high-dimensional pixel noise can be avoided.

[0088] Specifically, the action subsystem 120 can fuse the first visual features and the second visual features, and combine them with the second ontological perception state of the agent 200, in order to predict the actions of the agent 200 and generate specific motion control actions. In this way, the action subsystem 120 constitutes an inverse dynamic model in concept, which can calculate the dynamic physical transformation from the third time step to the fourth time step in a compact manifold space, thereby avoiding the processing of high-dimensional pixel noise.

[0089] In addition, since the feature spaces of the first visual feature and the second visual feature are consistent, the prediction subsystem 110 and the action subsystem 120 have a unified manifold space in terms of sample input, which ensures the strong consistency between the prediction subsystem 110 and the action subsystem 120 in the feature space.

[0090] In some cases, step 013 (fusing the first visual feature and the second visual feature to obtain the second fused feature) may include: Self-attention processing is applied to the first and second visual features; The first visual feature and the second visual feature after self-attention processing are subjected to cross-attention processing to obtain the second fused feature.

[0091] The processing method of the present invention can be implemented by the processing system 100 of the present invention. Specifically, please refer to... Figure 2 The action subsystem 120 can be used to perform self-attention processing on the first visual feature and the second visual feature; and to perform cross-attention processing on the first visual feature and the second visual feature after self-attention processing to obtain the second fused feature.

[0092] This increases the correlation between the first visual feature and the second visual feature.

[0093] Specifically, in Figure 6 In this system, the action subsystem 120 may include a self-attention module 121 and a cross-attention module 122. During pre-training, a first visual feature can be obtained through the prediction subsystem 110, and a second visual feature can be obtained through the second encoder 132. Then, the self-attention module 121 associates the first and second visual features, increasing the correlation between some features that are related to the first and second visual features. The associated first and second visual features are then input into the cross-attention module 122 for processing, thereby fusing the associated first and second visual features, which helps to further improve the correlation between the first and second visual features.

[0094] In some cases, step 01 (independent pre-training of the prediction subsystem 110 and the action subsystem 120 based on the acquired training samples) may include: The second ontological perception state of the agent 200 is input into the regression model of the action subsystem 120 to obtain vector data, which is used to represent the vector corresponding to the second ontological perception state in the feature space. Step 014 (inputting the second ontological perception state and the second fused features of the agent 200 into the action subsystem 120) may include: The vector data, the second ontology perception state, and the second fusion features are input into the action generation model 123 of the action subsystem 120.

[0095] The processing method of the present invention can be implemented by the processing system 100 of the present invention. Specifically, please refer to... Figure 2 The action subsystem 120 can be used to input the second ontology perception state of the agent 200 into the regression model of the action subsystem 120 to obtain vector data, which is used to represent the vector corresponding to the second ontology perception state in the feature space; and input the vector data, the second ontology perception state and the second fused features into the action generation model 123 of the action subsystem 120.

[0096] In this way, the second ontological perception state of the agent 200 can be easily combined with other inputs for processing.

[0097] Specifically, please combine Figure 6 ,exist Figure 6 In this context, the action subsystem 120 may include a second regression model 124. The second prophetic perception state characterizes the state information (such as joint angle information) of the agent 200 in physical space. Processing the second prophetic perception state through the second regression model 124 yields vector data corresponding to the vector information of the second prophetic perception state in the feature space; that is, the second prophetic perception state of the agent 200 is mapped between different spaces. Since the first and second visual features are also included in the vector information within the feature space, the inputs used for action prediction processing can be conveniently processed within a space of equal dimension.

[0098] In addition, the action generation model 123 may include a flow-matching-based DiT (DiffusionTransformer) network, a diffusion policy model (Diffusion Policy, U-Net structure), a multilayer perceptron (MLP), and a conventional Transformer decoder.

[0099] In some cases, step 05 (inputting the first predicted intent feature and the first prophetic perception state to the action subsystem 120 to obtain control information for controlling the agent 200) may include: The current visual features are fused with the first predicted intent features through the attention mechanism in the action subsystem 120 to obtain the first fused features; The first fusion feature is combined with the first ontology perception state and input into the action generation model 123 in the action subsystem 120 to generate control information.

[0100] The processing method of the present invention can be implemented by the processing system 100 of the present invention. Specifically, please refer to... Figure 2 The action subsystem 120 can be used to fuse the current visual features with the first predicted intent features through the attention mechanism in the action subsystem 120 to obtain the first fused features; combine the first fused features with the first ontology perception state and input them into the action generation model 123 in the action subsystem 120 to generate control information.

[0101] In this way, the actions of agent 200 can be deduced based on the prediction results.

[0102] Specifically, in the application scenario, since the first predicted intention feature is very close to the visual features corresponding to the predicted image that can be actually observed at the second time step, the current visual features and the first predicted intention feature are fused through the attention mechanism in the action subsystem 120. This allows the correlated features in the current visual features and the first predicted intention feature to be further associated, which is beneficial to clarifying the difference in motion state between the first image and the predicted image. Then, the first fused feature is combined with the first ontology perception state, so that the first ontology perception state is used as the initial state of the agent 200 at the first time step. The target state of the agent 200 at the second time step is determined by combining the first fused feature. Then, the action process of the agent 200 from the first time step to the second time step is deduced through the target state of the agent 200.

[0103] In some cases, the first prophetic perception state may include the joint angles of the agent 200. The first action block or the second action block may include the amount of change in joint angles. The amount of change in joint angles can be used to characterize the joint motion parameters required for the agent 200 to change from its current motion state to the motion state corresponding to the first action block or the second action block.

[0104] In this way, it is easy to achieve the effect of driving intelligent agents to control motion.

[0105] Specifically, please combine Figure 3 and Figure 4 With agent 200 in Figure 3 The state in the process is taken as the current motion state, and the agent 200 in... Figure 4 The state in the data is taken as the future motion state. After acquiring the first action block or the second action block, the agent 200 can obtain the expected change in the joint angle of the corresponding joint part at the corresponding time step (or, in other words, the direction and angle of rotation of the joint part). Then, within the corresponding time step, according to the corresponding change, the agent 200 controls the corresponding joint part to change the joint angle, so that the agent 200 can be in the expected motion state at the moment it is about to reach the next time step.

[0106] Furthermore, it can be understood that when H is 1, meaning the second time step is the next time step after the second time step, the agent 200 can drive the corresponding joint to rotate in the next time step according to the first action block. When H is greater than 1, meaning the second time step is several time steps after the second time step, the agent 200 can drive the corresponding joint to rotate in each time step according to all the second action blocks.

[0107] In some cases, step 014 (inputting the second ontological perception state and the second fused features of the agent 200 to the action subsystem 120) may include: At least one learnable second query term, a second ontology-aware state, and a second fusion feature are input into the action generation model 123 of the action subsystem 120. The second query term is used by the action subsystem 120 to query semantics and vision.

[0108] The processing method of the present invention can be implemented by the processing system 100 of the present invention. Specifically, please refer to... Figure 2 The action subsystem 120 can be used to input at least one learnable second query term, a second ontology-aware state, and a second fusion feature into the action generation model 123 of the action subsystem 120. The second query term is used for the action subsystem 120 to query semantics and vision.

[0109] This can enrich the common sense used to represent semantics and vision in the action subsystem 120, thereby improving the accuracy of action prediction processing.

[0110] Specifically, the second query term may include randomly initialized second continuous features. Multiple second continuous features can have continuously varying coordinate positions or vector angles within the feature space.

[0111] Please combine Figure 6 Specifically, the action generation model 123 can have a high-dimensional feature space containing a large number of high-dimensional vectors. These vectors have both a starting point and a direction within the feature space. The action generation model 123 performs action prediction on at least one learnable second query term, a second ontology-aware state, and a second fusion feature. This allows it to predict the meaning of words in the high-dimensional feature space that are close to the second ontology-aware state and the second fusion feature, thereby understanding the intent in the second ontology-aware state, the first visual feature, and the second visual feature.

[0112] Based on the above, by inputting the second query term into the action generation model 123, a large number of terms that can continuously change in coordinate position are added to the feature space of the action generation model 123, and / or a large number of terms that can continuously change in vector direction are added. This helps to improve the semantic and visual common sense richness of the action generation model 123, and thus can more accurately combine the second ontology perception state and the second fusion feature to predict actions.

[0113] Please refer to Figure 9 In some cases, step 02 (co-training the prediction subsystem 110 and the action subsystem 120) may include: 021: Connect the prediction subsystem 110 and the action subsystem 120 in series; 022: The concatenated prediction subsystem 110 and action subsystem 120 are jointly trained using the first sample image 410, wherein the loss function of the joint training includes action prediction loss and intention prediction loss.

[0114] The processing method of the present invention can be implemented by the processing system 100 of the present invention. Specifically, please refer to... Figure 2 The action subsystem 120 can be used to concatenate the prediction subsystem 110 and the action subsystem 120; the concatenated prediction subsystem 110 and action subsystem 120 are jointly trained using the first sample image 410, wherein the loss function of the joint training includes action prediction loss and intention prediction loss.

[0115] This prevents feature space collapse and gradient shortcuts from occurring during system training.

[0116] Specifically, please combine Figure 6 It can be understood that by concatenating the prediction subsystem 110 and the action subsystem 120, the action subsystem 120 directly uses the second predicted intent feature predicted by the prediction subsystem 110 instead of the second visual feature for action prediction. In this way, the action subsystem 120, as a lower-level module, can directly receive the output of the prediction subsystem 110, which is a higher-level module. In the loss function of joint training, since the features obtained through the prediction subsystem 110 are completely differentiable, the gradient generated by the action prediction loss can be smoothly backpropagated to the prediction subsystem 110. Simultaneously, this gradient is subject to the physical regularization constraint of the intent prediction loss, thereby preventing feature space collapse and gradient shortcuts in the overall system.

[0117] In addition, it can be understood that, compared with the VLA model in related technologies, the pre-decoupling preheating training of the present invention can effectively isolate the problem of severe gradient interference that is very easy to occur during joint training, which is conducive to ensuring the stability of end-to-end convergence and exhibiting higher robustness in cross-domain generalization and anti-interference tests of real robots (such as Agent 200).

[0118] Furthermore, since the present invention decouples the prediction subsystem 110 and the action subsystem 120, it is convenient to replace the specific architectural models constituting the prediction subsystem 110 and / or the action subsystem 120, thereby making it applicable to different multimodal large models that can have input token processing capabilities.

[0119] Please refer to Figure 10 A computer device 500 according to the present invention may include a memory 510 and a processor 520. The memory 510 may store a computer program. When the processor 520 executes the computer program, it can implement the steps of the processing method of the present invention.

[0120] For example, when a computer program is executed by processor 520, the processing methods that can be implemented include: 03: Acquire control commands, visual observation data collected at the first time step, and the first ontological perception state of the agent 200. The visual observation data includes the first image observed at the first time step. 04: Input the control command and visual observation data into the prediction subsystem 110 to obtain the first prediction intention feature of the predicted image expected to be observed at the second time step. The second time step is the Hth time step after the first time step, where H is a positive integer. 05: Input the first predicted intent feature and the first ontological perception state into the action subsystem 120 to obtain control information for controlling the agent 200. The control information is used to drive the joint movement of the agent 200 to realize the change in motion state between the first time step and the second time step indicated by the control command.

[0121] In the aforementioned computer device 500, the prediction subsystem 110 does not directly output action signals, but instead predicts the future characteristics of the agent 200. The action subsystem 120 can be used as an inverse dynamics model to deduce actions based on the future characteristics of the agent 200, thereby constructing a structured bottleneck of "intention-action". Compared with the traditional end-to-end VLA model, it can avoid the loss of generalization ability caused by directly fitting action noise, which is conducive to improving the generalization of the overall system composed of the prediction subsystem 110 and the action subsystem 120 when used for action control of the agent 200.

[0122] The present invention provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor 520, the computer program can implement the steps of the processing method of the present invention.

[0123] For example, when a computer program is executed by processor 520, the processing methods that can be implemented include: 03: Acquire control commands, visual observation data collected at the first time step, and the first ontological perception state of the agent 200. The visual observation data includes the first image observed at the first time step. 04: Input the control command and visual observation data into the prediction subsystem 110 to obtain the first prediction intention feature of the predicted image expected to be observed at the second time step. The second time step is the Hth time step after the first time step, where H is a positive integer. 05: Input the first predicted intent feature and the first proprioceptive state into the action subsystem 120 to obtain control information for controlling the agent 200. The first proprioceptive state includes the joint angles of the agent 200.

[0124] In the aforementioned computer-readable storage medium, the prediction subsystem 110 does not directly output action signals, but predicts the future characteristics of the agent 200. The action subsystem 120 can be used as an inverse dynamics model to deduce actions based on the future characteristics of the agent 200, thereby constructing a structured bottleneck of "intention-action". Compared with the traditional end-to-end VLA model, it can avoid the loss of generalization ability caused by directly fitting action noise, which is conducive to improving the generalization of the overall system composed of the prediction subsystem 110 and the action subsystem 120 when used for action control of the agent 200.

[0125] The computer-readable storage medium can be located in the computer device 500 or in other terminals. The computer device 500 can communicate with other terminals to obtain the corresponding program.

[0126] It is understood that computer-readable storage media can include: any entity or device capable of carrying a computer program, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc. Computer programs can include computer program code. Computer program code can be in the form of source code, object code, executable files, or certain intermediate forms, etc.

[0127] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0128] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a system including a processing module or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0129] Although embodiments of the present invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to the embodiments of the present invention without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A processing method for motion control of an intelligent agent, characterized in that, The processing method includes: The system acquires control commands, visual observation data collected at a first time step, and the first ontological perception state of the agent, wherein the visual observation data includes the first image observed at the first time step. The control command and the visual observation data are input into the prediction subsystem to obtain the first prediction intention feature of the predicted image expected to be observed at the second time step, where the second time step is the Hth time step after the first time step, and H is a positive integer. The first predicted intent feature and the first ontology perception state are input to the action subsystem to obtain control information for controlling the agent. The control information is used to drive the joint movement of the agent to realize the change in motion state between the first time step and the second time step indicated by the control command.

2. The processing method according to claim 1, characterized in that, The control information includes at least one of the following: The first action block corresponds to the motion state of the agent at the second time step; At least two second action blocks, each of which corresponds to the motion state of the agent in one of the H time steps following the first time step; And / or, The step of inputting the control command and the visual observation data into the prediction subsystem to obtain the first prediction intent feature of the expected observed prediction image at the second time step includes: The visual encoder in the prediction subsystem extracts the current visual features corresponding to the first image in the visual observation data; The control command and the current visual features are input into the visual language model of the prediction subsystem, and the first prediction intent feature is output. And / or, The processing method includes: The prediction subsystem and the action subsystem are independently pre-trained based on the acquired training samples. The training samples include a first sample image and a second sample image. The first sample image is the third image observed at the third time step, and the second sample image is the fourth image observed at the fourth time step. The fourth time step is the h-th time step after the third time step, where h is a positive integer. After the independent pre-training is completed, the prediction subsystem and the action subsystem are trained together.

3. The processing method according to claim 2, characterized in that, The step of inputting the first predicted intent feature and the first ontology perception state into the action subsystem to obtain control information for controlling the agent includes: The current visual features are fused with the first predicted intent features through the attention mechanism in the action subsystem to obtain the first fused feature; The first fusion feature is combined with the first ontology perception state and input into the action generation model in the action subsystem to generate the control information; And / or, The first ontological perception state includes the joint angles of the agent; The first action block or the second action block includes the change in the joint angle, which is used to characterize the joint motion parameters required for the agent to change from the current motion state to the motion state corresponding to the first action block or the second action block.

4. The processing method according to claim 2, characterized in that, The step of independently pre-training the prediction subsystem and the action subsystem based on the acquired training samples includes: The first visual feature and the language feature are input into the prediction subsystem to obtain the second prediction intention feature expected to be observed in the second sample image at the fourth time step. The first visual feature is obtained by encoding the first sample image, and the language feature corresponds to the training instruction issued in the training scenario. The parameters of the prediction subsystem are adjusted based on the second visual feature and the second prediction intention feature, wherein the second visual feature is obtained by encoding the second sample image; And / or, The step of inputting the first visual features and language features into the prediction subsystem includes: At least one learnable first query term, the first visual feature, and the linguistic feature are input into the prediction subsystem to obtain the second prediction intent feature, wherein the first query term is used by the prediction subsystem to query semantics and vision.

5. The processing method according to claim 4, characterized in that, Based on the second visual features and the second prediction intent features, the parameters of the prediction subsystem are adjusted, including: Obtain the mean squared error between the second predicted intent feature and the second visual feature; If the mean square error is greater than the preset error, the parameters of the prediction subsystem are adjusted.

6. The processing method according to claim 2, characterized in that, The step of independently pre-training the prediction subsystem and the action subsystem based on the acquired training samples includes: The first visual feature and the second visual feature are fused to obtain the second fused feature. The first visual feature and the second visual feature are obtained by encoding the first sample image and the second sample image, respectively. The feature spaces of the first visual feature and the second visual feature are consistent. The second ontology perception state of the agent and the second fused feature are input to the action subsystem to obtain the control information. The second ontology perception state is the ontology perception state of the agent in the training scenario.

7. The processing method according to claim 6, characterized in that, The process of fusing the first visual feature and the second visual feature to obtain the second fused feature includes: Self-attention processing is performed on the first visual feature and the second visual feature; The first visual feature and the second visual feature after self-attention processing are subjected to cross-attention processing to obtain the second fused feature; And / or, The step of independently pre-training the prediction subsystem and the action subsystem based on the acquired training samples further includes: The second prophecy perception state is input into the regression model of the action subsystem to obtain vector data, which is used to represent the vector corresponding to the second prophecy perception state in the feature space. The second ontological perception state of the agent and the second fused feature are input into the action subsystem, including: The vector data, the second ontology perception state, and the second fusion feature are input into the action generation model of the action subsystem.

8. The processing method according to claim 6 or 7, characterized in that, The second ontological perception state of the agent and the second fused feature are input into the action subsystem, including: At least one learnable second query term, the second ontology-aware state, and the second fusion feature are input into the action generation model of the action subsystem, wherein the second query term is used for the action subsystem to query semantics and vision.

9. The processing method according to claim 2, characterized in that, The joint training of the prediction subsystem and the action subsystem includes: The prediction subsystem and the action subsystem are connected in series; The concatenated prediction subsystem and action subsystem are jointly trained using the first sample image, wherein the loss function of the joint training includes action prediction loss and intention prediction loss.

10. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the processing method according to any one of claims 1 to 9.

Images