Robot operation method based on multi-modal learning

Abstract

The invention discloses a robot operation method based on multi-modal learning, and belongs to the field of multi-modal learning, and the method comprises the steps: collecting an object posture, a human operation video and a language instruction, and constructing multi-modal data; performing trajectory extraction and optimization on the data to obtain an operation trajectory meeting constraint conditions; generating a vision-language-action fusion strategy based on the trajectory, and forming multi-modal feature representation in combination with a prediction method of coordinate transformation and dynamic region guidance; in the action generation stage, future action condition probability distribution is modeled through a diffusion model Transform network, action and potential feature decoupling is achieved, and a control instruction capable of driving a robot is generated. The operation capability and task reliability of the robot in a complex environment are improved, and the method is suitable for industrial automation and service scenes.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of multimodal learning, and particularly relates to a robot operation method based on multimodal learning. BACKGROUND

[0002] Real-time operation decision is a key link to realize wide-view environmental perception and dexterous action execution of robots. Early research simplifies robot operation as a trajectory reproduction problem, which needs to rely on a large amount of artificial teaching data, and is difficult to meet the high generalization demand in complex scenarios. Although the classic motion planning method alleviates this problem to some extent, it is limited by the defects of slow planning speed and insufficient trajectory adaptability. In recent years, learning-based robot operation methods have developed rapidly, and have derived single-modal imitation learning, cross-modal feature alignment, language-conditioned strategy learning and diffusion generative strategy technical directions. Among them, the multimodal large model (MLM) models the scene information and task demand as a cross-modal implicit distribution through the Transformer architecture, which greatly promotes the technical breakthrough of operation learning.

[0003] However, the existing MLM method still faces significant challenges in the virtual-real fusion operation task of robots: first, it relies on large-scale offline labeled data, and the data acquisition and preprocessing cost is high; second, the computational overhead in the cross-modal alignment and action decoding process is too large, which seriously limits the feasibility of real-time operation. Although existing research attempts to reduce model complexity and data dependence to improve dynamic scene adaptability and universality, the operation robustness (such as environmental interference resistance) and real-time response speed still do not meet the requirements of practical applications. SUMMARY

[0004] To solve the above technical problems, the application provides a robot operation method based on multimodal learning, comprising:

[0005] According to the collected posture data of multiple objects and the corresponding human operation video, and in combination with the language instruction, a multimodal data is constructed;

[0006] According to the multimodal data, trajectory extraction and processing are performed to obtain an operation trajectory that meets the constraint condition;

[0007] According to the operation trajectory, a visual-language-action fusion strategy is constructed;

[0008] According to the fusion strategy, in combination with the coordinate transformation technology and the dynamic region guided world knowledge prediction method, spatial information and semantic information are integrated to form a multimodal feature representation;

[0009] According to the multi-modal feature representation, a diffusion model-based Transformer network is used to model a future action conditional probability distribution, and control instructions for driving the robot are generated.

[0010] Optionally, the multi-modal data construction comprises: aligning and synchronizing object images and corresponding operation videos under different perspectives; extracting key action frames and corresponding language instructions according to the aligned data; and constructing a unified multi-modal representation according to the key action frames and language instructions.

[0011] Optionally, the trajectory extraction and processing comprises: calculating an end pose sequence through forward kinematics according to a human hand joint angle sequence obtained by the motion capture glove; performing collision detection and stability analysis in a physics engine according to the pose sequence; and performing constraint optimization according to the analysis result to obtain an operation trajectory satisfying kinematic and dynamic constraints.

[0012] Optionally, after the operation trajectory is played back via a simulation environment, a multi-perspective visual sequence rendered is obtained, and the language instructions and action information are spliced to construct a fusion strategy sample.

[0013] Optionally, the fusion strategy generation comprises: generating an operation plan through a multi-modal Transformer network according to the operation trajectory and language instructions; and performing action selection and path planning by using spatial information and semantic information according to the operation plan.

[0014] Optionally, the world knowledge prediction method comprises: constructing an input sequence according to a local voxel grid and combining features output by a visual-language feature extractor; generating a latent representation via a Transformer network according to the input sequence; and forming a multi-modal compact encoding according to the latent representation.

[0015] Optionally, the diffusion model-based Transformer network modeling comprises: generating a future action sequence by using a DDIM reverse sampling method according to the latent representation; and mapping high-dimensional actions to a latent space and obtaining control signals in real time according to the future action sequence.

[0016] Optionally, the action execution comprises: generating joint control signals by combining robot body states according to a continuous action sequence output by the diffusion Transformer network; and converting the joint control signals into low-level execution instructions via a PID controller.

[0017] In another aspect, the present application also provides an electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.

[0018] In another aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method.

[0019] Compared with the prior art, the present application has the following advantages and technical effects:

[0020] The present application generates natural operation trajectories that conform to kinematic and dynamic constraints and fit human habits by collecting multi-modal data and performing trajectory extraction and physical-natural coupling optimization; forms compact multi-modal feature representation through unified modeling of vision, language and action, and world knowledge prediction method of coordinate transformation and dynamic area guidance; in the action generation stage, the effective decoupling of action features and potential task features is realized by using the Transformer network based on the diffusion model, and the real-time performance and robustness are improved by combining DDIM accelerated sampling; the generalization ability is enhanced through mixed training of simulation and real data and domain randomization method, thereby significantly improving the dexterous operation ability and task execution reliability of the robot in complex environments. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which form a part of this application, are intended to provide further understanding of the application and are incorporated herein in their entirety, and the illustrative embodiments thereof and their description serve to explain the present application. In the drawings:

[0022] Figure 1 The method flowchart of the embodiments of the present application is shown. DETAILED DESCRIPTION

[0023] It should be noted that the embodiments and features in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0025] Embodiment one

[0026] As Figure 1 shown, the present embodiment provides a robot operation method based on multi-modal learning, comprising:

[0027] Collecting various object poses and human operation videos, and constructing multi-modal data in combination with language instructions;

[0028] Trajectory extraction and processing are performed on the multi-modal data to obtain natural and reasonable operation trajectories that meet physical constraints and conform to human operation habits.

[0029] The operation trajectories are used to construct a unified visual-language-action fusion strategy.

[0030] In the strategy generation process, coordinate transformation technology and dynamic region guided world knowledge prediction method are combined to integrate spatial and semantic information, forming a compact multi-modal feature representation to accurately guide subsequent action execution.

[0031] In the action execution phase, a Transformer network based on a diffusion model is used to model the conditional probability distribution of future actions, effectively decoupling action features and potential task features, and generating control instructions that can directly drive the robot to execute.

[0032] Optionally, obtaining the multi-modal data includes:

[0033] Obtaining object images and corresponding operation videos from different perspectives, aligning and synchronizing the images and videos, extracting key action frames and corresponding language instructions, and constructing a unified multi-modal representation; the trajectory is corrected for physical constraints and optimized for naturalness to ensure that the operation trajectory meets the kinematics of the robot and environmental constraints during execution.

[0034] Optionally, generating the visual-language-action strategy includes:

[0035] Fusing the trajectory and language instructions to generate a task-oriented operation plan through a multi-modal Transformer or graph neural network; in the fusion process, spatial and semantic information are used to guide action selection and path planning, forming a compact and efficient operation representation.

[0036] Optionally, action execution includes:

[0037] A Transformer network based on a diffusion model predicts the conditional probability distribution of future actions, decoupling action features and potential task features; according to the prediction result, executable control instructions are generated to drive the robot to complete the corresponding operation task.

[0038] To achieve the above purpose, the embodiment also provides a robot operation system based on multi-modal learning, comprising:

[0039] A multi-modal data acquisition module is used to acquire object poses and human operation videos, and to construct multi-modal data in combination with language instructions.

[0040] A trajectory extraction and processing module is used to process the multi-modal data to obtain natural and reasonable operation trajectories that meet physical constraints and conform to human operation habits.

[0041] an operation strategy generation module, configured to generate a unified visual-language-action fusion strategy based on the trajectory, and integrate spatial and semantic information to form a compact multi-modal representation by combining a coordinate transformation technique and a world knowledge prediction method based on dynamic region guidance;

[0042] an action execution module, configured to model a future action conditional probability distribution by using a Transformer network based on a diffusion model, effectively decouple the action and the latent feature, and drive the robot to perform the operation.

[0043] Embodiment Two

[0044] The embodiment provides a robot operation method based on multi-modal learning, comprising:

[0045] According to the collected multi-object pose data and corresponding human operation video, and combined with the language instruction, a multi-modal data is constructed;

[0046] According to the multi-modal data, a trajectory extraction and processing is performed to obtain an operation trajectory satisfying a constraint condition;

[0047] According to the operation trajectory, a visual-language-action fusion strategy is constructed;

[0048] According to the fusion strategy, spatial information and semantic information are integrated to form a multi-modal feature representation by combining a coordinate transformation technique and a world knowledge prediction method based on dynamic region guidance;

[0049] According to the multi-modal feature representation, a future action conditional probability distribution is modeled by using a Transformer network based on a diffusion model to generate a control instruction for driving the robot.

[0050] A multi-view RGB-D camera and a wearable motion capture glove are used to synchronously collect a plurality of object poses and corresponding human operation videos, the motion capture glove is used to obtain a hand joint angle sequence, and the RGB-D camera is used to collect multi-view visual data during the operation; the accompanying natural language instruction is timestamped and aligned to construct a "visual-language-action" multi-modal data;

[0051] The multi-modal data is subjected to trajectory extraction and processing, the hand joint angle sequence collected by the glove is used to calculate the hand end position sequence by forward kinematics, and collision detection and stability analysis are performed in a physical engine, a natural and reasonable operation trajectory satisfying kinematic and dynamic constraints and conforming to human operation habits is obtained through constraint optimization; the optimized trajectory is played back in a simulation environment, a multi-view RGB-D sequence is rendered, and the language instruction and the action information are spliced into a unified "visual-language-action" fusion strategy sample;

[0052] In the strategy generation stage, a local coordinate system is established centering on the object dynamic area, a world knowledge prediction network is combined with the coordinate transformation technology and the dynamic area guidance, spatial geometry and semantic information are integrated, a compact multi-modal representation is formed, and the modeling capability for the dynamic changes of the environment is improved; in the action execution stage, a Transformer network based on a diffusion model is used to model the conditional probability distribution of T-step future actions, a DDIM acceleration sampling method is used to compress the high-dimensional action sequence to a latent space and decode in real time, the effective decoupling of the action features and the latent task features is realized, the continuous action instructions after decoding are sent to the robot joint controller, corresponding control signals are generated, and the dexterous operation task in a complex scene is completed.

[0053] The specific implementation of the trajectory physics-nature coupling optimization algorithm used in the embodiment is as follows:

[0054] In the actual collected human demonstration trajectory, there are often cases inconsistent with physical constraints, such as joint over-limit, collision or trajectory non-smoothness and the like. For this purpose, the present application proposes a physics-nature coupling optimization method, which ensures that the trajectory meets the constraints of robot execution while maintaining its naturalness and rationality. Specifically, let the original demonstration trajectory be τ, and the optimized trajectory be Lightweight optimization is performed through the following objective function:

[0055]

[0056] wherein, is a physical constraint penalty term (used to avoid collision and ensure stability), is a joint limit penalty term (ensuring that the trajectory does not exceed the range of the mechanical arm joint), is a trajectory smoothing term (ensuring trajectory continuity and naturalness), and λ and μ are balance coefficients. Through the above optimization, a natural and reasonable trajectory that meets both physical constraints and human operation habits can be obtained.

[0057] The specific implementation of the world knowledge prediction network used in the embodiment is as follows:

[0058] To enhance the modeling capability of the strategy for the task-related area, the embodiment introduces a world knowledge prediction network to jointly model the local spatial geometry and semantic features. Specifically, taking the local voxel grid V as the input, combining the multi-modal features f vl obtained by the visual-linguistic feature extractor, constructing an input sequence:

[0059] z=Transforrmer(f vl ;V);

[0060] Wherein, the Transformer network is used to capture the cross-modal context dependency, and the output latent representation z is a compact encoding of the "affordance-geometry" joint distribution, which can reflect the geometric shape of the object and the task-related semantic information at the same time. The latent code z will be directly used as the input of the diffusion strategy to guide the action prediction.

[0061] The specific implementation of the diffusion-Transformer action decoding method used in the embodiment is as follows:

[0062] In the action generation stage, the embodiment uses a Transformer network based on a diffusion model to model the future action sequence through a conditional probability distribution. The specific process is as follows: under the condition of a given latent code z, the action sequence is recovered by using DDIM (Denoising Diffusion Implicit Models) reverse sampling:

[0063]

[0064] Wherein, α t-1 is a noise scheduling coefficient, f θ is a diffusion network prediction function, ∈ θ is a noise estimator. Through the DDIM accelerated sampling method, a high-dimensional action sequence can be generated with only a small number of iterations, ensuring real-time performance and accuracy.

[0065] The specific implementation of the feature aggregation-action decoding integrated network used in the embodiment is as follows:

[0066] To further improve the inference efficiency and control accuracy, the embodiment integrates feature aggregation and action decoding into an end-to-end diffusion Transformer. Specifically, the latent code z obtained by the world knowledge prediction network is concatenated with the robot body state s as input, and after being encoded by the diffusion Transformer with residual connection, a continuous action sequence in the joint space is directly outputted:

[0067] α = DiffusionTransformer ([z; s]);

[0068] The generated action sequence is then converted into low-level control signals by a PID controller to drive the robot to execute. During the training process, mixed training is used with simulation replay data and real acquisition data, and the domain randomization method is used to reduce the gap between the simulation and the real environment, so as to ensure the stable operation of the strategy under the control frequency of 30Hz, and to have good generalization ability and robustness.

[0069] In another aspect, the embodiment further provides an electronic device, comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.

[0070] In another aspect, the embodiment further provides a computer readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement the method.

[0071] The above merely shows the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed by the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A robot operation method based on multimodal learning, characterized in that, include: Based on the collected object posture data and corresponding human operation videos, and combined with language commands, multimodal data is constructed. Based on the multimodal data, trajectory extraction and processing are performed to obtain the operation trajectory that meets the constraints. Based on the operation trajectory, a vision-language-action fusion strategy is constructed; Based on the aforementioned fusion strategy, combined with coordinate transformation technology and dynamic region-guided world knowledge prediction method, spatial information and semantic information are integrated to form a multimodal feature representation; Based on the multimodal feature representation, a Transformer network based on a diffusion model is used to model the conditional probability distribution of future actions, generating control commands for driving the robot.

2. The method according to claim 1, characterized in that, The construction of multimodal data includes: aligning and synchronizing object images and corresponding operation videos from different perspectives; extracting key action frames and corresponding language commands based on the aligned data; and constructing a unified multimodal representation based on the key action frames and language commands.

3. The method according to claim 1, characterized in that, The trajectory extraction and processing includes: calculating the end-effector pose sequence using forward kinematics based on the hand joint angle sequence obtained from the motion capture glove; performing collision detection and stability analysis in the physics engine based on the pose sequence; and performing constraint optimization based on the analysis results to obtain an operation trajectory that satisfies kinematic and dynamic constraints.

4. The method according to claim 1, characterized in that, The operation trajectory is replayed in the simulation environment, and then spliced ​​with language instructions and action information based on the rendered multi-view visual sequence to construct a fusion strategy sample.

5. The method according to claim 1, characterized in that, The generation of the fusion strategy includes: generating an operation plan through a multimodal Transformer network based on the operation trajectory and language instructions; and performing action selection and path planning based on the operation plan using spatial and semantic information.

6. The method according to claim 1, characterized in that, The world knowledge prediction method includes: constructing an input sequence based on a local voxel grid and combining features output by a visual-language feature extractor; generating a latent representation based on the input sequence via a Transformer network; and forming a multimodal compact code based on the latent representation.

7. The method according to claim 1, characterized in that, The Transformer network modeling based on the diffusion model includes: generating a sequence of future actions using the DDIM inverse sampling method based on the latent representation; mapping the high-dimensional actions to the latent space based on the future action sequence, and decoding the control signal in real time.

8. The method according to claim 1, characterized in that, The action execution includes: generating joint control signals based on the continuous action sequence output by the diffusion Transformer network and the robot's body state; and converting the joint control signals into low-level execution instructions via a PID controller.

9. An electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, characterized in that, When the processor executes the computing program, it implements the method of any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-8.

Images