Robot grabbing detection method based on visual language action multi-mode alignment strategy

By adopting visual language action multimodal alignment strategy and multi-action sorting task-oriented grasping strategy in the robot grasping system, the problem of grabbing complex tasks in the open world in the existing technology is solved, and a more efficient and robust robot grasping operation is achieved.

CN119526405AActive Publication Date: 2025-02-28JIANGSU UNIV OF SCI & TECH

Patent Information

Application Number
CN202411757782.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-12-03
Publication Date
2025-02-28
Estimated Expiration
2044-12-03

AI Technical Summary

Technical Problem

The existing task-oriented crawling methods are difficult to effectively respond to complex and changeable task needs in the open world, and the lack of task-oriented crawling strategies leads to greater limitations in practical applications.

Method used

A robot crawling detection method based on visual language action multimodal alignment strategy is proposed. By constructing a multimodal feature alignment and multi-action decision-making framework, multi-modal fusion of point cloud, language text and crawling actions is realized, and a task-oriented crawling strategy of multi-action sorting is adopted.

Benefits of technology

It improves the generalization ability and crawling success rate of robots in the open world, solves the problems of insufficient multimodal data fusion and insufficient task orientation, and achieves more efficient and robust crawling operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119526405A_ABST
    Figure CN119526405A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of robot grabbing control, and discloses a robot grabbing detection method based on a visual language action multi-mode alignment strategy, which comprises the following steps of: inputting a natural language instruction into a large language model, and generating three descriptions of an object category, a task and a grabbing action and an attention mask; performing channel attention optimization on the three types of data through a text encoder to obtain text encoding features; inputting three-dimensional point cloud data of an object, and generating point cloud coding features; inputting the point cloud data into a candidate grabbing posture coding module, and extracting grabbing posture coding features; multi-modal feature fusion is realized through a cross attention module, and fusion features are obtained; inputting the fusion features into a strategy module to obtain an evaluation value of each candidate grabbing action; and sorting all the candidate grabbing actions according to the evaluation value. According to the method, through alignment fusion of the point cloud, the language text and the grabbing posture, the task-oriented robot grabbing multi-action decision problem is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of robot grasping control, and in particular to a robot grasping detection method based on a visual language action multimodal alignment strategy. Background Art

[0002] With the rapid development of robotics technology, people have gradually placed more expectations on household robots, which are not only able to complete simple picking operations, but also need to understand and execute complex task instructions. For example, people hope that robots can assist in completing tasks that require high autonomy, such as tool use, item classification, and task operations, according to command instructions. This leads to the need for robots to grasp tools in a task-oriented manner, that is, task-oriented grasping. The core of this task is that the robot not only needs to be able to accurately complete the grasping action, but also needs to understand the purpose of the tool and its relationship with the task instructions according to the task requirements.

[0003] In task-oriented grasping tasks, robots need to have a deep understanding of instructions and be able to learn the connection between tasks and objects from limited examples. In addition, the learned knowledge needs to be generalized to new tasks, which places high demands on the understanding ability and adaptability of robot algorithms. However, existing task-oriented grasping methods have shortcomings in dealing with real-world tasks. First, early methods mostly rely on carefully designed scenes or predefined action dictionaries to simulate human grasping behavior. However, these methods are only applicable to specific closed scenes and are difficult to cope with complex and changing task requirements in the open world, which are greatly limited in practical applications. Inspired by human natural language processing capabilities, researchers gradually combine language text with visual information to enhance the robot's ability to perceive complex environments. However, most methods complete visual-language fusion through modules such as CLIP, but point cloud data and action data that are crucial to grasping tasks are often ignored, and there is a lack of truly usable multimodal fusion strategies. Secondly, existing methods mostly use direct splicing of point cloud and action data, and then splicing and fusion with language data encoding. However, this method fails to effectively solve the problem of uneven distribution between point cloud and motion data, and cannot fully explore the intrinsic connection between multimodal data. Finally, the current methods mainly adopt two types of grasping strategies: single action decision and grasping part segmentation. The former only focuses on a single grasping action, which is prone to task interruption due to planning failure in actual scenarios. The latter obtains multiple candidate actions by first segmenting the grasping part of the object and then performing grasping detection, thereby circumventing the shortcomings of single action decision. However, this method cannot incorporate task instructions or work scene restrictions into the grasping strategy, and lacks task orientation. Summary of the invention

[0004] In order to solve the problem that the above-mentioned prior art cannot incorporate task instructions or work scene restrictions into the grasping strategy and lacks task orientation, the present invention proposes a robot grasping detection method based on a visual language action multimodal alignment strategy, which improves the generalization ability and grasping success rate of the robot in performing tasks in the open world by constructing a multimodal feature alignment and multi-action decision framework.

[0005] The present invention is implemented by the following technical solution: comprising the following steps

[0006] S1. Input the natural language instructions into the pre-trained large language model LLM for parsing, generate three types of descriptions: object category description, task description, and grasping description, and input the three types of descriptions together with the natural language instructions into the pre-trained BERT model for processing to obtain high-dimensional text features and attention masks;

[0007] S2, input the 3D point cloud data of the object, extract the geometric and structural features through the pre-trained PointNet++ point cloud encoder, and generate the point cloud embedding F p ;

[0008] S3, input the point cloud data into the pre-trained FGC-GraspNet module to generate a set of candidate grasping postures g i ; The candidate grasping posture g i Expand to action feature X g , and extract the grasping feature F through the multi-layer perceptron MLP g ;

[0009] S4: Use the text encoder model to optimize the channel attention of the three types of descriptions and natural language instructions obtained in step S1 to obtain the language text feature F txt ;

[0010] S5. Based on the captured feature F g , a modality alignment module composed of a multi-layer perceptron is used to align the point cloud features F p and language text features F txt Perform dimension alignment to obtain multimodal features;

[0011] S6. According to the cross attention mechanism, the action feature F g Set as query Query, point cloud feature F p Set to key Key, language text feature F txt Set to the value Value, and realize the fusion of multimodal features through the cross attention module to obtain the fusion feature F fusion ;

[0012] S7, the fusion feature F obtained in step S6 fusionInput strategy module, extract important information of grasping features through multi-layer perceptron MLP, and obtain evaluation value S of each candidate grasping action through Sigmoid activation function;

[0013] S8. Sort all candidate grasping actions according to the evaluation value S of each candidate grasping action obtained in step S7, output the sorted action list, control the robot to execute the optimal grasping action, and fall back to the suboptimal action if the grasping fails.

[0014] As a further preferred embodiment, the pre-trained PointNet++ point cloud encoder in step S2 includes a multi-scale point cloud feature extraction layer MSG and a feature fusion layer SA, and the formula for generating point cloud embedding is F p =SA(MSG(X p , dim=0)), through the formula F p =Align(squeeze(F p ,dim=1),dim=1,num=N) to embed the point cloud F p to expand.

[0015] As a further preference, the specific steps of step S3 are as follows:

[0016] S31, each candidate grasping posture g of the [R, T] homogeneous matrix of size 4×4 i Expanded into one-dimensional feature X g =[x,y,z,q0,q1,q2,q3];

[0017] Among them, x, y, z represent the translation relationship, x, y, z = T tr ;q0,q1,q2,q3 represent the quaternion of the rotation matrix;

[0018] S32, the expanded grasping posture feature X g Input the capture encoder composed of a multi-layer perceptron MLP and get F g , the formula is as follows:

[0019] F g =MLP(X g , dim=3)

[0020] Where, X g It is the action feature after expansion.

[0021] As a further preference, the specific steps of step S4 are as follows:

[0022] S41, the text encoder model encodes the three types of descriptions and natural language instructions obtained in step S1 to generate a task description paragraph Object Class Description Section Capture description paragraphs and language instructions word embeddings;

[0023] S42, use the channel attention mechanism SE module to weight the text features to obtain the language text feature F txt , the formula is as follows:

[0024] F txt =SE(stack(F td ,F od ,F gd ,F I ,dim=1))

[0025] Among them, stack will F td , F od , F gd , F I Splice into a unified feature representation of multiple channels;

[0026] S43, the fused language features F txt To perform channel copy operation, the formula is as follows:

[0027] F txt =Align(squeeze(F txt ,dim=1),dim=1,num=N)

[0028] Where N represents the candidate grasping posture set g i The number of grasping postures in the squeeze function is determined by removing redundant dimensions from the language features and then replicating them by channel.

[0029] As a further preferred embodiment, the fusion feature F is obtained in step S6 fusion The formula is as follows:

[0030] F fusion =cross_attention(Q=F g ,K=F p ,V=F txt )

[0031] Among them, the action feature F g For query Q, point cloud feature F p is the key K, the language text feature F txt is the value V. As a further preferred embodiment, the formula for obtaining the evaluation value S of each candidate grasping action is as follows:

[0032] S = Sigmoid(MLP(F fusion ),dim=-1).

[0033] As a further preferred embodiment, the point cloud embedding in step S2

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. The present invention realizes the multi-modal alignment and fusion of point cloud, language text and grasping action, fully explores the intrinsic connection between the modalities, and solves the problem of insufficient multi-modal data fusion in the existing methods;

[0036] 2. This paper designs a task-oriented grasping strategy based on multi-action sequencing to improve the grasping efficiency and robustness of the robot in complex task scenarios and avoid task interruption caused by failure of single action planning;

[0037] 3. Provide a semantically enhanced data organization method, which uses the grasping semantic knowledge generated by the large language model to guide the robot to dynamically adjust the grasping strategy and enhance task orientation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the task-oriented grasping and detection process in the method of the present invention.

[0039] Figure 2 The present invention is a flow chart of language feature encoding in the method of the present invention.

[0040] Figure 3 This is a block diagram of the multimodal feature alignment and fusion module in the method of the present invention.

[0041] Figure 4 Schematic diagram of a real robot comparison experiment for task-oriented grasping in the method of the present invention. DETAILED DESCRIPTION

[0042] Advantages and features of the present invention will be illustrated and explained by the following non-limiting description of preferred embodiments thereof, which are given by way of example only with reference to the accompanying drawings.

[0043] like Figure 1 As shown, the present invention provides a robot grasping detection method based on a visual language action multimodal alignment strategy, comprising the following steps:

[0044] Step 1: Input the natural language instructions into the pre-trained large language model LLM for parsing, generate three types of descriptions: object category description, task description, and grasping description, and input the three types of descriptions together with the natural language instructions into the pre-trained BERT model for processing to obtain high-dimensional text features and attention masks.

[0045] Among them, the object category description extracts the target object category information in the natural language instruction, such as "tool" or "bottle"; the task description parses the semantic knowledge of the task goal, such as "cutting" or "pouring water"; the grasping description generates grasping action information related to the task, such as "pay attention to the handle of the knife". Subsequently, the above three descriptions and the natural language instructions are sent to the pre-trained text encoder for processing to generate high-dimensional text features for subsequent multimodal fusion.

[0046] Step 2: Input the 3D point cloud data of the object, extract the geometric and structural features through the pre-trained PointNet++ point cloud encoder, and generate a point cloud embedding The F p is the point cloud feature. The pre-trained PointNet++ point cloud encoder includes a multi-scale point cloud feature extraction layer MSG and a feature fusion layer SA. The formula for generating point cloud embedding is as follows:

[0047]

[0048] Then we use the point cloud alignment strategy to embed F p To expand, it will be copied according to the channel dimension to align it with the candidate grasping posture data. The alignment formula is as follows:

[0049] F p =Align(squeeze(F p ,dim=1),dim=1,num=N)

[0050] Where N is the set of candidate grasping postures g i The number of grip positions in the middle, squeeze is F p Add dimension, Align is copied according to channel dimension.

[0051] Step 3: Input the point cloud data into the pre-trained FGC-GraspNet module to generate a set of candidate grasping postures g i ; The candidate grasping posture g i Expand to action feature X g , and extract the grasping feature F through the multi-layer perceptron MLP g .

[0052] Step 31: Substitute the candidate grasping posture g of each [R,T] homogeneous matrix of size 4×4 i Expanded into one-dimensional feature X g =[x,y,z,q0,q1,q2,q3];

[0053] Among them, x, y, z represent the translation relationship, x, y, z = T tr ; q0, q1, q2, q3 represent the quaternion of the rotation matrix.

[0054] Step 32: Expand the grasping posture feature X g Input the capture encoder composed of a multi-layer perceptron MLP and get F g , the formula is as follows:

[0055] F g =MLP(X g , dim=3)

[0056] Different from the previous method of directly splicing point cloud data with grasping data, the present invention performs i Separate processing is performed to better align point cloud data and text features. Specifically, the 4×4 [R, T] homogeneous matrix of each candidate grasping posture is first expanded into a one-dimensional feature X g =[x,y,z,q0,q1,q2,q3]. Among them, x,y,z represent the translation relationship, the relationship is as follows:

[0057] x,y,z=T tr

[0058] Among them, q0, q1, q2, q3 represent the quaternion of the rotation matrix. g Send it to the capture encoder composed of a multi-layer perceptron MLP to get F g :

[0059] F g =MLP(X g , dim=3)

[0060] By extracting features through MLP, we can ensure that F g The dimension of the middle channel does not change, which facilitates the subsequent feature fusion process.

[0061] Step 4: Use the text encoder model to perform attention masking and channel attention optimization on the three types of descriptions and natural language instructions obtained in step 1 to obtain the language text feature F txt .

[0062] Step 41: The text encoder model encodes the three types of descriptions and natural language instructions obtained in step 1 to generate a task description paragraph. Object Class Description Section Capture description paragraphs and language instructions where td, od, gd, and I represent the maximum length of each language sequence (zero-padded). During training, the language encoder is frozen.

[0063] Step 42: Use the channel attention mechanism SE module to perform weighted processing on the text features to obtain the language text feature F txt , the formula is as follows:

[0064] F txt =SE(stack(F td ,F od ,F gd ,F I ,dim=1))

[0065] Among them, stack will F td , F od , F gd , F I The SE module dynamically adjusts the contribution of each text feature in the fusion by learning the importance weights.

[0066] Step 43: fusion of language features F txt To perform channel copy operation, the formula is as follows:

[0067] F txt =Align(squeeze(F txt ,dim=1),dim=1,num=N)

[0068] Where N represents the candidate grasping posture set g i The number of grasping postures in the squeeze function is determined by removing redundant dimensions from the language features and then replicating them by channel.

[0069] like Figure 2 As shown, in order to effectively link the three concepts of task, object and grasping posture, the present invention uses a pre-trained BERT model to parse natural language instructions, generates multiple semantic text features as input, and optimizes feature fusion through a channel attention mechanism.

[0070] Feeding language instructions into the BERT model pre-trained on a large-scale text data corpus eliminates the need to train the encoder from scratch, thus avoiding the consumption of a lot of time and computing resources.

[0071] In order to make full use of multiple semantic text features as semantic knowledge, the channel attention mechanism SE module is used to weight the text features. p and the grasping feature F g Alignment processing, the fused language text features F txt Perform channel duplication operation. Through the above steps, the language text feature F generated txt Can be combined with point cloud features F p and the grasping feature F gMaintain consistency to ensure the alignment and effectiveness of multimodal features in the subsequent fusion process.

[0072] Step 5: According to the captured feature F g , a modality alignment module composed of a multi-layer perceptron is used to align the point cloud features F p and language text features F txt Perform dimension alignment operations to obtain multimodal features.

[0073] Step 6: According to the cross attention mechanism, the action feature F g Set as query Query, point cloud feature F p Set to key Key, language text feature F txt Set to the value Value, and realize the fusion of multimodal features through the cross attention module to obtain the fusion feature F fusion The formula is as follows:

[0074] F fusion =cross_attention(Q=F g ,K=F p ,V=F txt )

[0075] Among them, the action feature F g For query Q, point cloud feature F p is the key K, the language text feature F txt is the value V.

[0076] like Figure 3 As shown, the point cloud feature F is p 、Language text features F txt and action feature F g Perform dynamic fusion to achieve deep connection and feature alignment between multiple modalities. The cross attention formula is as follows:

[0077]

[0078] First, the action feature F g Set as query Query, point cloud feature F p Set to key Key, language text feature F txt Set to value Value. These three types of features are consistent in channels and dimensions to ensure that multimodal features can be aligned accordingly.

[0079] Then, input the three types of features into the cross-attention module and calculate the fused cross-attention feature representation:

[0080] F fusion =cross_attention(Q=F g ,K=Fp ,V=F txt )

[0081] This module is used to align multiple grasping postures with a single object and text semantics to obtain the multimodal fusion feature F. fusion .

[0082] Through the above steps, the present invention realizes the point cloud feature F p 、Language text features F txt and action feature F g The dynamic fusion of multimodal features provides basic support for task-oriented grasping action planning.

[0083] Step 7: The fusion feature F obtained in step 6 fusion The input strategy module extracts important information of the grasping features through the multi-layer perceptron MLP, and obtains the evaluation value S of each candidate grasping action through the Sigmoid activation function.

[0084] The cross-attention features are input into the strategy module, and all candidate grasping actions are evaluated based on the fused multimodal features. Specifically, the fused feature F fusion The input strategy module consists of three fully connected layers MLP, each of which contains one-dimensional batch normalization BatchNorm and ReLU activation function to extract important information of grasping features. The evaluation value S of each candidate grasping action is calculated by sigmoid activation function:

[0085] S = Sigmoid(MLP(F fusion ), dim=-1)

[0086] Step 8: Sort all candidate grasping actions according to the evaluation value S of each candidate grasping action obtained in step 7, output the sorted action list, control the robot to perform the optimal grasping action, and fall back to the suboptimal action if the grasping fails.

[0087] Example 1

[0088] This embodiment takes the instruction "use the cup to drink." as an example to describe and explain the detection method of the present invention.

[0089] Step 1. Input the natural language instruction "use the cup to drink." to the pre-trained large language model LLM for parsing. This example uses the OpenAI GPT-3.5-turbo large language model output to generate the following three key descriptions.Object category description: "A household cup typically has a cylindrical shape with acircularbase and a uniform cross-section along its height, making it a three-dimensional object. Its geometric properties can be described in terms of its dimensions such as diameter, height, and volume, and is often used inmathematics to study concepts related to solid geometry and measurements.", Task description: "Household objects, such as cups, glasses, and mugs, provide a physical container for holding various liquids, facilitating the intake of beverages for hydration and nourishment. Additionally, items like straws and lids helpcontrol the flow of liquid and maintain the temperature of drinks to optimize the drinking experience.", Grab description: "The handle is the most suitable part for a robot hand to grasp as it is designed specifically forgripping. Additionally, gripping the rim of the cup may also provide astablegrip for the robot hand due to the relatively flat and sturdy surface." The above three types of descriptions are fed into the frozen BERT model together with natural language instructions to obtain description features and attention masks.

[0090] Step 2: Input the RGBD image collected by the depth camera, and output the 3D point cloud data through the point cloud sampling generation algorithm, with the shape of [4095,3]. Send the point cloud data to the PointNet++ module, which consists of two layers of multi-scale point cloud feature extraction layers MSG and one layer of feature fusion layer SA, to obtain the point cloud feature F p , with a shape of [1,768].

[0091] Step 3: Generate a set of candidate grasping postures g from the point cloud data through the pre-trained FGC-GraspNet module i , with a shape of [25,4,4], representing 25 candidate grasping actions. i Expand to action feature X g , with a shape of [25,1,7]. Input the grasp encoder composed of a multi-layer perceptron MLP to generate the grasp feature F g , with a shape of [25,768].

[0092] Step 4: Use the text encoder model to output the following features for the three types of descriptions and natural language instructions obtained in step 1: Task description paragraph Object Class Description Section Capture description paragraphs and language instructions The word embedding of BERT includes two parts: description features and attention mask. The description features and attention mask in the text word embedding processed by BERT are sent to the Max-pooling module to remove the text padding part, so that the model pays more attention to the effective information. Then, the four sets of text features are sent to the channel attention module to obtain the language text feature F txt , with a shape of [1,768].

[0093] Step 5: According to the action feature F g , for the point cloud feature F p and language text features F txt Perform dimension alignment operations to obtain three sets of multimodal features with a shape of [25,768].

[0094] Step 6: Set the action feature F g Set as query Query, point cloud feature F p Set to key Key, language text feature F txt Set to value Value, and send them to the cross attention module together to get the fusion feature F fusion , with a shape of [25,768].

[0095] Step 7: Fusion feature F fusionIt is sent to the strategy module composed of a multi-layer perceptron to obtain the evaluation value S of each candidate grasping action, which has a shape of [25,1].

[0096] Step 8: Sort the candidate grasping actions according to their evaluation value S, and give priority to actions that meet the task instructions.

[0097] like Figure 4 As shown in the figure, the model is trained on the public dataset LA-TaskGrasp, which contains point cloud data of common household objects and their corresponding task description texts. In terms of hardware platform, the Elite EC66 robot is used to complete the experimental deployment and verify the grasping success rate and efficiency of this method in actual scenarios.

[0098] In addition to the above embodiments, the present invention may also have other implementation modes. Any technical solutions formed by equivalent replacement or equivalent transformation shall fall within the protection scope required by the present invention.

Claims

1. A robot grasping detection method based on a visual language action multimodal alignment strategy, characterized by: The following steps are included: S1. Input the natural language instructions into the pre-trained large language model LLM for parsing, generate three types of descriptions: object category description, task description, and grasping description, and input the three types of descriptions together with the natural language instructions into the pre-trained BERT model for processing to obtain high-dimensional text features; S2, input the 3D point cloud data of the object, extract the geometric and structural features through the pre-trained PointNet++ point cloud encoder, and generate the point cloud embedding F p ; S3, input the point cloud data into the pre-trained FGC-GraspNet module to generate a set of candidate grasping postures g i ; The candidate grasping posture g i Expand to action feature X g , and extract the grasping feature F through the multi-layer perceptron MLP g ; S4: Use the text encoder model to optimize the channel attention of the three types of descriptions and natural language instructions obtained in step S1 to obtain the language text feature F txt ; S5. Based on the captured feature F g , a modality alignment module composed of a multi-layer perceptron is used to align the point cloud features F p and language text features F txt Perform dimension alignment to obtain multimodal features; S6. According to the cross attention mechanism, the action feature F g Set as query Query, point cloud feature F p Set to key Key, language text feature F txt Set to the value Value, and realize the fusion of multimodal features through the cross attention module to obtain the fusion feature F fusion ; S7, the fusion feature F obtained in step S6 fusion Input strategy module, extract important information of grasping features through multi-layer perceptron MLP, and obtain evaluation value S of each candidate grasping action through Sigmoid activation function; S8. Sort all candidate grasping actions according to the evaluation value S of each candidate grasping action obtained in step S7, output the sorted action list, control the robot to execute the optimal grasping action, and fall back to the suboptimal action if the grasping fails.

2. The robot grasping detection method based on the visual language action multimodal alignment strategy according to claim 1 is characterized in that: The pre-trained PointNet++ point cloud encoder in step S2 includes a multi-scale point cloud feature extraction layer MSG and a feature fusion layer SA. The formula for generating point cloud embedding is F p =SA(MSG(X p , dim=0)), through the formula F p =Align(squeeze(F p ,dim=1),dim=1,num=N) to embed the point cloud F p to expand.

3. The robot grasping detection method based on the visual language action multimodal alignment strategy according to claim 2 is characterized in that: The specific steps of step S3 are as follows: S31, each candidate grasping posture g of the [R, T] homogeneous matrix of size 4×4 i Expanded into one-dimensional feature X g =[x,y,z,q0,q1,q2,q3]; Among them, x, y, z represent the translation relationship, x, y, z = T tr ;q0,q1,q2,q3 represent the quaternion of the rotation matrix; S32, the expanded grasping posture feature X g Input the capture encoder composed of a multi-layer perceptron MLP and get F g , the formula is as follows: F g =MLP(X g ,dim=3) Where, X g It is the action feature after expansion.

4. The robot grasping detection method based on the visual language action multimodal alignment strategy according to claim 3 is characterized in that: The specific steps of step S4 are as follows: S41, the text encoder model encodes the three types of descriptions and natural language instructions obtained in step S1 to generate a task description paragraph Object Class Description Section Capture description paragraphs and language instructions word embeddings; S42, use the channel attention mechanism SE module to weight the text features to obtain the language text feature F txt , the formula is as follows: F txt =SE(stack(F td ,F od ,F gd ,F I ,dim=1)) Among them, stack will F td , F od , F gd , F I Splice into a unified feature representation of multiple channels; S43, the fused language features F txt To perform channel copy operation, the formula is as follows: F txt =Align(squeeze(F txt ,dim=1),dim=1,num=N) Where N represents the candidate grasping posture set g i The number of grasping postures in the squeeze function is determined by removing redundant dimensions from the language features and then replicating them by channel.

5. The robot grasping detection method based on the visual language action multimodal alignment strategy according to claim 4 is characterized in that: In step S6, the fusion feature F is obtained. fusion The formula is as follows: F fusion =cross_attention(Q=F g ,K=F p ,V=F txt ) Among them, the action feature F g For query Q, point cloud feature F p is the key K, the language text feature F txt is the value V.

6. The robot grasping detection method based on the visual language action multimodal alignment strategy according to claim 5 is characterized in that: The formula for obtaining the evaluation value S of each candidate grasping action is as follows: S=Sigmoid(MLP(F fusion ),dim=-1)。 7. The robot grasping detection method based on the visual language action multimodal alignment strategy according to claim 6 is characterized in that: The point cloud embedding in step S2

Citation Information

Patent Citations

  • Robot self-adaption fuzzy control method and system

    CN110497418A

  • Multimodal combined gesture action generation method based on diffusion model

    CN117636481A

  • Robot grabbing method and device combining vision and language instruction guidance

    CN118721192A

  • Multi-modal transfer learning method for large language model

    CN118863010A

  • Single stream multi-level alignment for vision-language pretraining

    US20230281963A1

Cited By

  • Method and system for guiding three-dimensional point cloud robot based on natural language

    CN119927932A

  • Multi-specification target object grabbing method based on large model

    CN120244978A

  • Mechanical arm grabbing method based on reference segmentation and spatial perception

    CN120901933A

  • Decision-making method and device guided by multi-modal semantic map, equipment and medium

    CN120952167A

  • Mechanical arm grabbing posture detection method based on diffusion model and language driving

    CN120973915A