Robot control method and device, electronic equipment and medium

Abstract

The embodiment of the invention relates to the technical field of robot control, and provides a robot control method and device, electronic equipment and a medium, and the method comprises the steps: obtaining the multi-mode perception information of a target robot, and generating a visual language condition vector; inputting the visual language condition vector and the multi-modal perception information into an average velocity field neural network for single-step reasoning to obtain a target control action instruction; controlling the target robot based on the target control action instruction; wherein the average velocity field neural network is trained by the following steps: acquiring sample multi-modal perception information to generate a sample visual language condition vector; calculating an expected average velocity field according to an average flow matching formula; and training the neural network based on the sample visual language condition vector, and training the average velocity field of the neural network to regress the expected average velocity field to obtain an average velocity field neural network. The planning trajectory of the complete action sequence is generated in one step, and the model reasoning efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a robot control method, device, electronic device, and medium. Background Technology

[0002] With the continuous advancement of embodied intelligence technology, robots are poised for widespread application in elderly care, healthcare, and industry. Currently, the challenge of real-time, efficient control of embodied robots remains a key focus in robot product development, particularly in highly dynamic and uncertain environments. Finding a universal, highly adaptable, and efficient real-time control method can significantly improve the safety, efficiency, and economic benefits of robots.

[0003] The patent application, published on March 11, 2025, with publication number CN119897865A, and entitled "A Method and Related Apparatus for Generating Robot Actions," describes a method that uses a multimodal information fusion module to acquire fused features and a conditional flow matching method for action generation. However, this method cannot effectively address the delay problem of multi-step denoising in robot control reasoning, especially for problems requiring operational control in dynamic environments. It also fails to consider the impact of the robot's physical and spatiotemporal constraints on control.

[0004] The patent application, published on May 20, 2025, with publication number CN120176714A, entitled "A Visual-Language Navigation Method for Mobile Robots Based on Diffusion Strategy," describes a method that integrates visual, linguistic, local historical displacement, and global displacement information to construct a conditional diffusion Transformer decoding module. This module then decodes the noise vector based on the conditional vector, applying it to a multi-step denoising process to output multiple candidate multi-step navigation trajectories. A contrastive learning method is then used to construct a navigation trajectory discriminator, efficiently and autonomously determining the matching scores between candidate trajectories and target points, enabling autonomous navigation of the robot in complex and dynamic environments. However, this method still involves a multi-step denoising iterative process, affecting the robot's real-time inference control.

[0005] The patent application, published on February 21, 2025, with publication number CN 119973991A, entitled "A Multimodal Embossed Intelligent Robot Control Device," describes a method that uses a brain-like decision-making model to analyze the action target and current external environment information to make decisions about the robot's behavior, thereby converting commands into low-level control instructions. However, the brain-like large model in this method suffers from excessive inference latency, making it difficult for the low-level servo control unit to cope with the complex and changing external environment.

[0006] In summary, the current challenges of large-model-based robot control are: (1) real-time inference performance needs to be considered so that the robot can work at high frequency and high precision; (2) cross-platform generalization and multimodal collaboration capabilities need to be considered so that heterogeneous robot systems can uniformly process multimodal inputs and generalize to unknown tasks and environments; (3) physical constraints need to be considered so that the executed actions can naturally satisfy the laws of kinematics and dynamics. Summary of the Invention

[0007] This invention provides a robot control method, device, electronic device, and medium to address the shortcomings of existing technologies, such as poor real-time performance, low accuracy, and weak generalization of robots in dynamic environments due to multi-step denoising iterations and neglect of physical constraints. It enables the planning of a complete action sequence to be generated in a single step, significantly improving the inference efficiency of large visual-language-action models.

[0008] This invention provides a robot control method, comprising: Acquire multimodal perception information of the target robot, and generate a visual language conditional vector based on the multimodal perception information; The visual language condition vector and the multimodal perception information are input into the average velocity field neural network for single-step reasoning to obtain the target control action command; The target robot is controlled based on the target control action commands; The average velocity field neural network is trained through the following steps: Acquire multimodal perception information of samples, and generate sample visual language conditional vectors based on the multimodal perception information of samples; Calculate the desired average velocity field based on the average flow matching formula; The neural network is trained based on the sample visual language condition vector, and the average velocity field of the trained neural network regresses the expected average velocity field to obtain the average velocity field neural network.

[0009] In one possible implementation, the method further includes: The visual language conditional vector and the multimodal perception information are input into the average velocity field neural network, and the complete motion trajectory is obtained by single-step forward integration through the average velocity field neural network. The complete motion trajectory is mapped to a feasible set that satisfies the physical constraints of the target robot by using a projection operator, resulting in a trajectory table that can be directly used for servo drive. Target control action instructions are generated based on the trajectory table.

[0010] In one possible implementation, the method further includes: Based on the visual language conditional vector and the multimodal perception information, a single-step estimated action block is obtained through single-step reasoning using a first formula, which is: ; in, This is a single-step estimated action block obtained from inference based on the average velocity field neural network. For embedded representation, For the initial noise distribution, For average velocity field neural networks; The complete motion trajectory is obtained by combining the single-step estimated action blocks.

[0011] In one possible implementation, the method further includes: Based on the second formula, a projection operator is used to correct the projection of each single-step estimated action block in the complete motion trajectory. The second formula is: ; in, For the corrected single-step estimated action block, For projection operators; The complete motion trajectory after projection correction is mapped to a feasible set that satisfies the physical constraints of the target robot, resulting in a trajectory table that can be directly used for servo drive.

[0012] In one possible implementation, the method further includes: Construct the identity equation for the average velocity field based on the displacement integral; The identity equation of the average velocity field is simplified, and the integral is differentiated to calculate the desired average velocity field.

[0013] In one possible implementation, the method further includes: A neural network is constructed, and the Jacobian matrix and directional derivative are calculated based on the average velocity field identity equation to obtain the average velocity field of the neural network. A training loss function is constructed based on the average velocity field; The neural network is trained based on the sample visual language condition vector and the training loss function. The average velocity field is trained to regress the expected average velocity field, resulting in an average velocity field neural network. The average velocity field neural network is capable of single-step reasoning to obtain control action commands.

[0014] The present invention also provides a robot control device, comprising the following modules: The acquisition and generation module is used to acquire multimodal perception information of the target robot and generate visual language conditional vectors based on the multimodal perception information. The reasoning module is used to input the visual language condition vector and the multimodal perception information into the average velocity field neural network for single-step reasoning to obtain the target control action command; The control module is used to control the target robot based on the target control action commands; The average velocity field neural network is trained through the following steps: acquiring multimodal perception information of samples and generating sample visual language condition vectors based on the multimodal perception information of samples; calculating the expected average velocity field according to the average flow matching formula; training the neural network based on the sample visual language condition vectors, and training the average velocity field of the neural network to regress the expected average velocity field, thereby obtaining the average velocity field neural network.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the robot control method as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot control method as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the robot control method as described above.

[0018] The robot control method, device, electronic device, and medium provided by this invention acquire multimodal perception information of a target robot and generate a visual language condition vector based on the multimodal perception information; input the visual language condition vector and the multimodal perception information into an average velocity field neural network for single-step inference to obtain a target control action command; and control the target robot based on the target control action command. The average velocity field neural network is trained through the following steps: acquiring sample multimodal perception information and generating sample visual language condition vectors based on the sample multimodal perception information; calculating the desired average velocity field according to the average flow matching formula; training the neural network based on the sample visual language condition vectors, and training the average velocity field of the neural network to regress the desired average velocity field, thus obtaining the average velocity field neural network. Compared to existing technologies that suffer from poor real-time performance, low accuracy, and weak generalization in dynamic environments due to multi-step denoising iterations and neglect of physical constraints, this solution achieves the planning of a complete action sequence in a single step, significantly improving the inference efficiency of the visual language action model. It also enables high-frequency and high-precision action control, allowing the robot to perform high-precision continuous actions in dynamic environments. Furthermore, it achieves cross-platform generalization and multimodal collaboration capabilities, enabling heterogeneous robot systems to uniformly process multimodal inputs and generalize to unknown tasks and environments. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a system framework diagram of the robot control method provided by the present invention.

[0021] Figure 2 This is a flowchart illustrating the robot control method provided by the present invention.

[0022] Figure 3 This is a schematic diagram of the average velocity field construction and average velocity field differential calculation process provided by the present invention.

[0023] Figure 4 This is a schematic diagram of the training and inference process of the average velocity field neural network provided by the present invention.

[0024] Figure 5 This is a schematic diagram of the robot control device provided by the present invention.

[0025] Figure 6This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0028] Figure 1 This is a system framework diagram of the robot control method provided by the present invention, such as... Figure 1 As shown, this method, based on a large visual language model and matching the average velocity field flow, includes the following steps: Step 1: The Vision-Language Model (VLM) receives multimodal input, including perceptual information such as language, multi-view visual images, and point clouds, and generates a visual-language conditional vector.

[0029] Step 2: Calculate the desired average velocity field according to the average flow matching formula, and train the neural network to regress the average velocity field to the desired average velocity field.

[0030] Step 3: Based on the visual language feature vector, ontological perception information, and the average velocity field of the neural network, obtain the control action command through single-step reasoning.

[0031] Specifically, Figure 2 This is a flowchart illustrating the robot control method provided by the present invention, as shown below. Figure 2 As shown, the method includes the following: S21. Obtain multimodal perception information of the target robot, and generate a visual language conditional vector based on the multimodal perception information.

[0032] The target robot collects multimodal perception information through its onboard sensor system, including language commands (such as voice or text commands), multi-view visual images (such as image data from multiple cameras), and point cloud data (such as three-dimensional spatial information from LiDAR or depth cameras).

[0033] This multimodal perception information is input into a large visual language model, which has the ability to fuse and understand multimodal data, extract semantic information from it, and generate a visual language conditional vector. This conditional vector is essentially a high-level semantic representation of the current task and environment, which will be used in subsequent control decision-making processes to ensure that the robot's actions can understand and respond to complex natural language commands and visual scenes.

[0034] S22. Input the visual language condition vector and the multimodal perception information into the average velocity field neural network for single-step reasoning to obtain the target control action command.

[0035] S23. Control the target robot based on the target control action command.

[0036] The average velocity field neural network is trained through the following steps: Acquire multimodal perception information of samples and generate sample visual language condition vectors based on the sample multimodal perception information; calculate the expected average velocity field according to the average flow matching formula; train the neural network based on the sample visual language condition vectors, and regress the expected average velocity field of the trained neural network to obtain the average velocity field neural network.

[0037] Specifically, an identity equation for the average velocity field is constructed based on the displacement integral. This equation is then simplified by differentiating the integral to calculate the desired average velocity field. A neural network is then constructed, and the Jacobian matrix and directional derivative are calculated based on the average velocity field identity equation to obtain the average velocity field of the neural network. A training loss function is constructed based on the average velocity field. The neural network is trained using the sample visual language condition vector and the training loss function, regressing the average velocity field to the desired average velocity field, resulting in an average velocity field neural network. This average velocity field neural network is capable of single-step reasoning to obtain control action commands.

[0038] Specifically, such as Figure 3 As shown, the true average velocity field based on the displacement integral definition is as follows: in, For interpolating noise motion trajectories, For real action blocks, These represent the start time, the current stream matching time, and the visual language condition vector from the VLM, respectively. Let be the true average velocity field and the instantaneous velocity field, respectively. τ is an auxiliary infinitesimal introduced when transforming the integral equation into a differential equation, used to derive the instantaneous rate of change of the average velocity field through the limiting process. dτ is the differential form of τ.

[0039] By making a simple transformation of the above formula and differentiating the integral, we obtain the following average flow identity: Following the above method, such as Figure 4 As shown, the specific form of constructing the average velocity field neural network is as follows: According to the average flow identity: The following can be derived: in, For Jacobian matrices, and For the directional derivative, It satisfies the basic ODE equation.

[0040] Furthermore, since the true average velocity field is unknown, a neural network is needed to estimate the Jacobian matrix to obtain the approximate expected average velocity field formula as follows: The training loss function is then constructed as follows: Here, a stop gradient operation is applied to the approximate expected average velocity field to avoid higher-order optimization.

[0041] Furthermore, based on visual language feature vectors, ontology perception information, and mean-field neural networks, control action commands are obtained through single-step inference, and the following formula is executed: in, This is a single-step estimated action block obtained from inference based on the average velocity field neural network. For embedded representation, For the initial noise distribution, It is an average velocity field neural network.

[0042] To further ensure that the generated action blocks satisfy the robot's physical constraints, the generated actions need to be projected and corrected as follows: in, For the corrected single-step estimated action block, For the projection operator, a typical modification is to use quadratic programming, as follows: After receiving the corrected target control motion command, the command is sent to the target robot's execution system (such as a servo driver or joint controller) to drive the robot to complete the corresponding action. For systems such as robotic arms, the motion command may need to be converted into joint angle or torque commands through inverse kinematics solving; for mobile robots, the motion command may directly correspond to velocity or acceleration control. The entire control process is conducted in a closed-loop manner, meaning that the robot continuously acquires new sensory information and updates its control strategy in real time while performing actions, thereby ensuring high-precision, high-response-speed trajectory tracking and operation tasks in dynamic environments. This method generates a complete action sequence through single-step inference, avoiding the delay of traditional multi-step denoising processes and significantly improving the real-time control performance of robots in complex dynamic environments.

[0043] The robot control method provided by this invention acquires multimodal perception information of a target robot and generates a visual language condition vector based on the multimodal perception information; inputs the visual language condition vector and the multimodal perception information into an average velocity field neural network for single-step inference to obtain a target control action command; and controls the target robot based on the target control action command. The average velocity field neural network is trained through the following steps: acquiring sample multimodal perception information and generating sample visual language condition vectors based on the sample multimodal perception information; calculating the desired average velocity field according to the average flow matching formula; training the neural network based on the sample visual language condition vectors, and training the average velocity field of the neural network to regress the desired average velocity field, thus obtaining the average velocity field neural network. Compared to existing technologies that suffer from poor real-time performance, low accuracy, and weak generalization in dynamic environments due to multi-step denoising iterations and neglect of physical constraints, this method achieves the planning of a complete action sequence in a single step, significantly improving the inference efficiency of the visual language action model. It also enables high-frequency and high-precision action control, allowing the robot to perform high-precision continuous actions in dynamic environments. Furthermore, it achieves cross-platform generalization and multimodal collaboration capabilities, enabling heterogeneous robot systems to uniformly process multimodal inputs and generalize to unknown tasks and environments.

[0044] The robot control method provided by this invention is applicable to trajectory tracking and dexterous manipulation tasks of embodied intelligent systems such as non-holonomic wheeled robots and robotic arms. First, it receives multimodal input, including language, multi-view visual images, and point cloud perception information, based on a large visual language model, generating a visual language condition vector. Then, it calculates the desired average velocity field according to the average flow matching formula and trains a neural network to regress the desired average velocity field. Next, it calculates the desired average velocity field again using the average flow matching formula and trains the neural network to regress the desired average velocity field. Subsequently, based on the visual language feature vector, ontological perception information, and the neural network's average velocity field function, it performs single-step inference to obtain control action commands and performs photographic corrections to satisfy physical constraints. Finally, it sends the planned trajectory directly to the servo driver or solves for joint commands through inverse kinematics, achieving closed-loop adjustment and tracking. This method effectively improves upon traditional multi-step denoising methods for robot flow matching or diffusion models, enabling the generation of a complete action sequence planned trajectory in a single step. It significantly improves the inference efficiency of the large visual language action model and has significant application value for real-time operation tasks of robots working in complex dynamic environments. It is suitable for trajectory tracking and dexterous operation tasks of incomplete wheeled robots, robotic arms and other embodied intelligent systems.

[0045] The robot control device provided by the present invention is described below. The robot control device described below and the robot control method described above can be referred to in correspondence.

[0046] Figure 5 This is a schematic diagram of the robot control device provided by the present invention, specifically including: The acquisition and generation module 501 is used to acquire multimodal perception information of the target robot and generate a visual language conditional vector based on the multimodal perception information. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0047] The inference module 502 is used to input the visual language condition vector and the multimodal perception information into the average velocity field neural network for single-step inference to obtain the target control action command. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0048] The control module 503 is used to control the target robot based on the target control action command. For detailed explanation, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0049] The average velocity field neural network is trained through the following steps: acquiring multimodal perception information of samples and generating sample visual language conditional vectors based on the multimodal perception information; calculating the desired average velocity field according to the average flow matching formula; training the neural network based on the sample visual language conditional vectors, and training the average velocity field of the neural network to regress the desired average velocity field, thereby obtaining the average velocity field neural network. For detailed explanations, please refer to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0050] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a robot control method, which includes: acquiring multimodal perception information of a target robot and generating a visual language condition vector based on the multimodal perception information; inputting the visual language condition vector and the multimodal perception information into an average velocity field neural network for single-step inference to obtain a target control action command; and controlling the target robot based on the target control action command. The average velocity field neural network is trained through the following steps: acquiring sample multimodal perception information and generating sample visual language condition vectors based on the sample multimodal perception information; calculating the desired average velocity field according to the average flow matching formula; training the neural network based on the sample visual language condition vectors, and training the average velocity field of the neural network to regress the desired average velocity field, thereby obtaining the average velocity field neural network.

[0051] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0052] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the robot control method provided by the above methods. The method includes: acquiring multimodal perception information of a target robot and generating a visual language condition vector based on the multimodal perception information; inputting the visual language condition vector and the multimodal perception information into an average velocity field neural network for single-step inference to obtain a target control action command; and controlling the target robot based on the target control action command. The average velocity field neural network is trained through the following steps: acquiring sample multimodal perception information and generating sample visual language condition vectors based on the sample multimodal perception information; calculating a desired average velocity field according to the average flow matching formula; training the neural network based on the sample visual language condition vectors, and training the average velocity field of the neural network to regress the desired average velocity field to obtain the average velocity field neural network.

[0053] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the robot control method provided by the above methods. The method includes: acquiring multimodal perception information of a target robot and generating a visual language condition vector based on the multimodal perception information; inputting the visual language condition vector and the multimodal perception information into an average velocity field neural network for single-step inference to obtain a target control action command; and controlling the target robot based on the target control action command. The average velocity field neural network is trained through the following steps: acquiring sample multimodal perception information and generating sample visual language condition vectors based on the sample multimodal perception information; calculating a desired average velocity field according to the average flow matching formula; training the neural network based on the sample visual language condition vectors, training the average velocity field of the neural network to regress the desired average velocity field, thereby obtaining the average velocity field neural network.

[0054] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0055] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot control method, characterized in that, include: Acquire multimodal perception information of the target robot, and generate a visual language conditional vector based on the multimodal perception information; The visual language condition vector and the multimodal perception information are input into the average velocity field neural network for single-step reasoning to obtain the target control action command; The target robot is controlled based on the target control action commands; The average velocity field neural network is trained through the following steps: Acquire multimodal perception information of samples, and generate sample visual language conditional vectors based on the multimodal perception information of samples; Calculate the desired average velocity field based on the average flow matching formula; The neural network is trained based on the sample visual language condition vector, and the average velocity field of the trained neural network regresses the expected average velocity field to obtain the average velocity field neural network.

2. The method according to claim 1, characterized in that, The step of inputting the visual language conditional vector and the multimodal perception information into the average velocity field neural network for single-step inference to obtain the target control action command includes: The visual language conditional vector and the multimodal perception information are input into the average velocity field neural network, and the complete motion trajectory is obtained by single-step forward integration through the average velocity field neural network. The complete motion trajectory is mapped to a feasible set that satisfies the physical constraints of the target robot by using a projection operator, resulting in a trajectory table that can be directly used for servo drive. Target control action instructions are generated based on the trajectory table.

3. The method according to claim 2, characterized in that, The process of obtaining the complete motion trajectory block through single-step forward integration of the average velocity field neural network includes: Based on the visual language conditional vector and the multimodal perception information, a single-step estimated action block is obtained through single-step reasoning using a first formula, which is: ; in, This is a single-step estimated action block obtained from inference based on the average velocity field neural network. For embedded representation, For the initial noise distribution, For average velocity field neural networks; The complete motion trajectory is obtained by combining the single-step estimated action blocks.

4. The method according to claim 2, characterized in that, The process of mapping the complete motion trajectory to a feasible set that satisfies the physical constraints of the target robot using a projection operator to obtain a trajectory table that can be directly used for servo drive includes: Based on the second formula, a projection operator is used to correct the projection of each single-step estimated action block in the complete motion trajectory. The second formula is: ; in, For the corrected single-step estimated action block, For projection operators; The complete motion trajectory after projection correction is mapped to a feasible set that satisfies the physical constraints of the target robot, resulting in a trajectory table that can be directly used for servo drive.

5. The method according to any one of claims 1-4, characterized in that, The calculation of the desired average velocity field based on the average flow matching formula includes: Construct the identity equation for the average velocity field based on the displacement integral; The identity equation of the average velocity field is simplified, and the integral is differentiated to calculate the desired average velocity field.

6. The method according to any one of claims 1-4, characterized in that, The step of training the neural network based on the sample visual language conditional vector, and training the average velocity field of the neural network to regress the expected average velocity field to obtain the average velocity field neural network, includes: A neural network is constructed, and the Jacobian matrix and directional derivative are calculated based on the average velocity field identity equation to obtain the average velocity field of the neural network. A training loss function is constructed based on the average velocity field; The neural network is trained based on the sample visual language condition vector and the training loss function. The average velocity field is trained to regress the expected average velocity field, resulting in an average velocity field neural network. The average velocity field neural network is capable of single-step reasoning to obtain control action commands.

7. A robot control device, characterized in that, include: The acquisition and generation module is used to acquire multimodal perception information of the target robot and generate visual language conditional vectors based on the multimodal perception information. The reasoning module is used to input the visual language condition vector and the multimodal perception information into the average velocity field neural network for single-step reasoning to obtain the target control action command; The control module is used to control the target robot based on the target control action commands; The average velocity field neural network is trained through the following steps: acquiring multimodal perception information of samples and generating sample visual language condition vectors based on the multimodal perception information of samples; calculating the expected average velocity field according to the average flow matching formula; The neural network is trained based on the sample visual language condition vector, and the average velocity field of the trained neural network regresses the expected average velocity field to obtain the average velocity field neural network.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the robot control method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot control method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the robot control method as described in any one of claims 1 to 6.

Images