Visual language action large model design method for enhancing spatial perception ability

Abstract

The invention discloses a visual language action large model design method for enhancing spatial perception ability, which comprises the following steps of: acquiring a plurality of frames of task scene images, inputting the images into a two-dimensional image encoder and a visual geometry guidance Transformer VGGT encoder, extracting image features and performing time sequence spatial embedding, encoding a natural language instruction into language embedding representation, and extracting a visual language action large model. Image features and language embedding are fused through a cross attention mechanism, a pre-training large model is input to generate cross-modal representation, the fusion features are input into an action expert module to output an action control sequence in combination with robot body state information, and a robot is driven to execute an operation task. Therefore, damage of additional information to an original pre-training model is avoided, and compared with an original combined structure of a pre-training visual language model and an action expert, utilization of multi-view picture information is enhanced, so that higher understanding ability on space depth is achieved in the task execution process, the task success rate is increased, and the task execution efficiency is improved. And a more efficient and more robust robot sensing and decision-making integrated system is realized.

Description

Technical Field

[0001] The present invention relates to the field of robot control and decision-making models, and specifically to a method for designing a large visual language action model for enhancing spatial perception capabilities. Background Art

[0002] With breakthroughs in image understanding and language interaction, visual language models (VLMs) have achieved superior performance in tasks such as visual question answering, image description, and video scene understanding. However, most existing VLMs have significant deficiencies in spatial reasoning capabilities.

[0003] On the one hand, they have difficulty accurately understanding the relative distance and direction between objects, making them unable to effectively perform robotic tasks that require clear depth perception and geometric understanding. In particular, when they work with downstream action experts to combine semantic information with the robot's state information to output low-level action commands, the success rate is often low.

[0004] On the other hand, the visual geometry-anchored Transformer VGGT can infer rich three-dimensional attributes such as camera parameters, depth maps, point clouds, etc. from multiple frames of images through a single forward propagation, significantly improving the efficiency and accuracy of spatial understanding and 3D reconstruction tasks. In the field of robotics, existing studies have used different methods to input three-dimensional point cloud information into the network to improve spatial perception capabilities, but most methods still rely on additional hardware sensors, such as depth cameras, or perform triangulation reconstruction based on multiple cameras. Summary of the Invention

[0005] The present invention aims to solve the technical problems in the above-mentioned technologies at least to some extent.

[0006] To this end, the present invention discloses a method for designing a large visual language action model for enhancing spatial perception ability, comprising:

[0007] S1. Obtain multiple frames of task scene images, input the multiple frames of images into a two-dimensional image encoder and a visual geometry guided Transformer (VGGT) encoder, extract image features and temporal spatial embedding, where the temporal spatial embedding includes depth perception features and geometric motion information;

[0008] S2. Segment and encode the natural language instructions to obtain language embedding representation;

[0009] S3, embedding the image features obtained in step S1 into the language embedding representation obtained in step S2, fusing them through a cross-attention mechanism, and inputting them into a pre-trained large model to generate a cross-modal fusion representation;

[0010] S4, combining the robot's body state information, inputting the fusion features generated in step S3 into the action expert module, and outputting the action control sequence;

[0011] S5. Drive the robot to execute the operation task corresponding to the language instruction according to the action control sequence.

[0012] The method for designing a large visual language action model for enhancing spatial perception disclosed in the present invention has at least the following technical effects:

[0013] (1) This invention improves the spatial geometry perception capability of multi-frame fusion, and uses the native ability of the VGGT encoder to directly obtain three-dimensional geometric information such as camera parameters, depth map and spatial point cloud from multi-frame visual input, achieving efficient spatial understanding without complex post-processing optimization;

[0014] (2) The present invention integrates the visual features extracted by the two-dimensional image encoder and the VGGT encoder, language instructions, and robot state information into the same visual language pre-training model through a cross-modal cross-attention mechanism, ensuring that spatial, semantic, and state information interact collaboratively in a unified context, thereby improving fusion efficiency and expression consistency.

[0015] (3) This invention avoids deep modification of the original visual language pre-training model architecture. By adding an external geometric encoder and a cross-attention module, the semantic reasoning ability of the pre-trained VLM is preserved while achieving enhanced spatial ability.

[0016] (4) Compared with the combination of pre-trained VLM and action experts, the proposed system makes full use of multi-frame image spatial information and has a stronger understanding of depth, posture, and geometric changes, thereby improving the success rate of robot execution and environmental generalization ability;

[0017] In summary, the visual language action large model design method for enhancing spatial perception ability disclosed in the present invention can, on the one hand, work together through the geometric encoder and the cross-attention module to link action experts to realize a general architecture that integrates the robot from spatial perception to action execution, and is suitable for multimodal robot control scenarios in open environments and diversified tasks. On the other hand, it has the advantage of enhancing spatial understanding ability without destroying the structure of the pre-trained model. Combined with the VGGT geometric modeling module and the multimodal attention fusion strategy, it significantly improves the robot's spatial understanding and execution success rate in complex tasks while maintaining the original semantic reasoning ability.

[0018] In addition, the method for designing a large visual language action model for enhancing spatial perception disclosed in the present invention may also have the following additional technical features:

[0019] In one embodiment of the present invention, the VGGT encoder extracts depth perception features and geometric motion information from a multi-frame image sequence through a single-frame attention and inter-frame cross-attention mechanism.

[0020] In one embodiment of the present invention, the natural language instruction includes a target object description or an action target description.

[0021] In one embodiment of the present invention, the two-dimensional image encoder adopts a Vision Transformer architecture, and the VGGT encoder is a Transformer structure including a single-frame and inter-frame cross-attention mechanism.

[0022] In one embodiment of the present invention, the pre-trained large model is CLIPBLIP or a combination thereof.

[0023] In one embodiment of the present invention, the action expert module is a multi-layer perceptron diffusion denoising model or a Transformer decoder.

[0024] In one embodiment of the present invention, the body state information of the robot includes at least one of a joint angle, an end effector position, and a state, wherein the end effector includes but is not limited to a gripper or a dexterous hand.

[0025] In one embodiment of the present invention, the fusion method in step S3 is cross attention fusion or multi-layer perceptron fusion.

[0026] In one embodiment of the present invention, the action control sequence is used to control a real robot or an agent in a virtual environment to perform actions including but not limited to grasping, moving or placing operations.

[0027] Additional contents and advantages of the present invention will be given in the following description or can be understood through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The technical solutions and beneficial effects of the present invention will become apparent and easily understood from the following contents in conjunction with the accompanying drawings, in which:

[0029] Figure 1 This is a system framework diagram of the method for designing a large visual language action model for enhancing spatial perception capability according to the present invention;

[0030] Figure 2 This is a specific implementation flow chart of the method for designing a large visual language action model for enhancing spatial perception ability of the present invention. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0032] The following describes the method for designing a large visual language action model for enhancing spatial perception disclosed by the present invention with reference to the accompanying drawings. Figure 1 and Figure 2 As shown:

[0033] It should be noted that the method for designing a large visual language action model for enhancing spatial perception disclosed in the present invention addresses issues such as insufficient spatial understanding and high target execution failure rate. The system includes hardware components such as a camera, an end effector, a robotic arm, and a control computing unit.

[0034] The method for designing a large visual language action model for enhancing spatial perception includes:

[0035] S1. Obtain multi-frame task scene images and input them into a 2D image encoder and a visual geometry-guided Transformer (VGGT) encoder, respectively, to extract image features and temporal spatial embeddings. The temporal spatial embeddings include depth perception features and geometric motion information.

[0036] S2. Segment and encode the natural language instructions to obtain language embedding representation;

[0037] S3: Embed the image features obtained in step S1 into the language embedding representation obtained in step S2, fuse them through the cross-attention mechanism, and input them into the pre-trained large model to generate a cross-modal fusion representation;

[0038] S4, combining the robot's body state information, inputting the fusion features generated in step S3 into the action expert module, and outputting the action control sequence;

[0039] S5. Drive the robot to execute the operation task corresponding to the language instruction according to the action control sequence.

[0040] The VGGT encoder extracts depth-aware features and geometric motion information from multi-frame image sequences through single-frame attention and inter-frame cross-attention mechanisms.

[0041] The natural language instruction includes a target object description or an action target description.

[0042] The two-dimensional image encoder adopts the Vision Transformer architecture, and the VGGT encoder is a Transformer structure that includes a single-frame and inter-frame cross-attention mechanism.

[0043] The pre-trained large model is CLIPBLIP or its combination.

[0044] The action expert module is a multi-layer perceptron diffusion denoising model or a Transformer decoder.

[0045] The robot's body state information includes at least one of a joint angle, an end effector position, and a state, wherein the end effector may be a gripper or a dexterous hand.

[0046] The fusion method in step S3 is cross attention fusion or multi-layer perceptron fusion.

[0047] Action control sequences are used to control real robots or agents in virtual environments to perform grasping, moving, placing, or other operations.

[0048] Specifically:

[0049] Step 1: Initialization: Given language instructions, the language instructions may include natural language task descriptions such as "grab the red bottle" and "put the Coke into the basket";

[0050] Step 2: The system obtains images from the environment camera, the robot arm camera, and the robot body joint status data at a fixed frequency as perception input;

[0051] Step 3: The image data is input into the VGGT encoder and the 2D image encoder respectively. The VGGT encoder is used to extract the 3D structure information and geometric anchor embedding in the multi-frame image, and the 2D image encoder is used to capture the semantic features of the static image to obtain the visual feature sequence.

[0052] Step 4: The language instructions are processed by the word segmenter to form a language embedding vector, which is preliminarily spliced ​​or aligned with the visual features output in step 3 in the fusion module to form a unified modality input;

[0053] Step 5: The fused multimodal input enters the pre-trained vision-language model, undergoes deep fusion through a multi-layer Transformer cross-attention mechanism, and outputs a compressed semantic-spatial representation.

[0054] Step 6: The representation output from step 5 and the current state of the robot input from step 2 are input into the action expert module. The action expert module can be a multi-layer perceptron, a diffusion denoising process, or a Transformer decoder, and outputs a control vector or action sequence.

[0055] Step 7: Send the motion control signal output from step 6 to the robot controller to complete basic operations such as grasping, moving, and placing, and execute the target behavior described in the task language;

[0056] Step 8: The system executes until it determines that the task is completed or the upper limit of the number of attempts is reached.

[0057] As a possibility, in one embodiment of the present invention, in step 3, the two-dimensional image encoder is a VisionTransformer;

[0058] For other details, please refer to the above contents, which will not be described in detail in this embodiment.

[0059] As a possibility, in one embodiment of the present invention, in step 4, the fusion module refers to a cross-attention mechanism, where the query is the image encoding information output by the Vision Transformer, and the key-value pair is the spatial encoding information output by the VGGT;

[0060] For other details, please refer to the above contents, which will not be described in detail in this embodiment.

[0061] As a possibility, in one embodiment of the present invention, in step 7, the robot is a Xarm6 robotic arm, and the controller is Xarm-Python-SDK;

[0062] For other details, please refer to the above contents, which will not be described in detail in this embodiment.

[0063] In summary, the method for designing a large visual language action model for enhancing spatial perception ability disclosed in the present invention can, on the one hand, work together through the geometric encoder and the cross-attention module to link action experts to realize a general architecture that integrates the robot from spatial perception to action execution, which is suitable for multimodal robot control scenarios in open environments and diversified tasks. On the other hand, it has the advantage of enhancing spatial understanding ability without destroying the structure of the pre-trained model. Combined with the VGGT geometric modeling module and the multimodal attention fusion strategy, it significantly improves the robot's spatial understanding and execution success rate in complex tasks while maintaining the original semantic reasoning ability.

[0064] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for designing a large visual language action model to enhance spatial perception, characterized by: include: S1. Obtain multiple frames of task scene images, input the multiple frames of images into a two-dimensional image encoder and a visual geometry guided Transformer (VGGT) encoder, extract image features and temporal spatial embedding, where the temporal spatial embedding includes depth perception features and geometric motion information; S2. Segment and encode the natural language instructions to obtain language embedding representation; S3, embedding the image features obtained in step S1 into the language embedding representation obtained in step S2, fusing them through a cross-attention mechanism, and inputting them into a pre-trained large model to generate a cross-modal fusion representation; S4, combining the robot's body state information, inputting the fusion features generated in step S3 into the action expert module, and outputting the action control sequence; S5. Drive the robot to execute the operation task corresponding to the language instruction according to the action control sequence.

2. The method for designing a large visual language action model for enhancing spatial perception according to claim 1, characterized in that: The VGGT encoder extracts depth perception features and geometric motion information from multi-frame image sequences through single-frame attention and inter-frame cross-attention mechanisms.

3. The method for designing a large visual language action model for enhancing spatial perception according to claim 1, characterized in that: The natural language instruction includes a target object description or an action target description.

4. The method for designing a large visual language action model for enhancing spatial perception according to claim 1, wherein: The two-dimensional image encoder adopts the Vision Transformer architecture, and the VGGT encoder is a Transformer structure that includes a single-frame and inter-frame cross-attention mechanism.

5. The method for designing a large visual language action model for enhancing spatial perception according to claim 1, wherein: The pre-trained large model is CLIPBLIP or a combination thereof.

6. The method for designing a large visual language action model for enhancing spatial perception according to claim 1, wherein: The action expert module is a multi-layer perceptron diffusion denoising model or a Transformer decoder.

7. The method for designing a large visual language action model for enhancing spatial perception according to claim 1, wherein: The robot's body state information includes at least one of a joint angle, an end effector position, and a state, wherein the end effector includes but is not limited to a gripper or a dexterous hand.

8. The method for designing a large visual language action model for enhancing spatial perception according to claim 1, wherein: The fusion method in step S3 is cross attention fusion or multi-layer perceptron fusion.

9. The method for designing a large visual language action model for enhancing spatial perception according to claim 1, wherein: The action control sequence is used to control the execution of a real robot or an intelligent agent in a virtual environment, including but not limited to grasping, moving or placing operations.

Images