A large model feature fusion hashing self-attention method for robotic operations

Abstract

The invention discloses a large model feature fusion hash self-attention method for robot operation, obtains teaching data of multiple complex operation tasks in the target operation environment; collects multimodal information in the operation scene; constructs a visual-text refining and alignment module based on a large model, inputs the sound signal and visual signal in the multimodal information, obtains the final extracted visual features and the final predicted text features; uses a feature extractor to convert the final obtained visual features and text features into the same dimension of features, uses the designed hash interactive robot skill learning module to perform feature interaction and feature extraction, and predicts the corresponding robot's action; trains the hash interactive robot skill learning module based on the teaching data and the preset loss function, and is used to predict the robot's action according to real-time visual and text input. Improve the robot's autonomous learning, skill generalization and precise control capabilities in a complex processing operation environment.

Description

Technical Field

[0001] The present invention belongs to the field of robotics technology, and in particular relates to a large model feature fusion hashing self-attention method for robotic operations. Background Art

[0002] The complexity of robot operation scenarios continues to increase, and the demand for intelligent production operations is becoming increasingly urgent. In modern production operations, the application of robot systems can significantly improve experimental efficiency, accuracy and safety. However, current robot operating systems still have the following problems when facing complex operation tasks:

[0003] The ability to learn and generalize skills is insufficient, making it difficult to adapt to new tasks; the operating accuracy is difficult to meet processing requirements; the environmental adaptability is poor, making it difficult to cope with dynamic changes in complex scenes; the real-time response capability is insufficient, making it impossible to accurately control time-sensitive experimental processes; the system generally lacks flexibility and is difficult to meet diverse experimental needs.

[0004] These problems have severely restricted the development and application of intelligent robot systems, and there is an urgent need for an innovative method that can comprehensively improve the autonomous learning and skill generalization of multi-robot systems in complex operating environments. Summary of the invention

[0005] In view of the above technical problems, the present invention provides a large model feature fusion hash self-attention method for robot operations.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] A large model feature fusion hashing self-attention method for robot operation, the method comprising the following steps:

[0008] S100: Acquire teaching data of various complex operation tasks in the target operation environment;

[0009] S200: Collects and integrates visual, acoustic and textual multi-dimensional information to fully capture multi-modal information in the robot operation scene;

[0010] S300: constructing a large-model-based visual-text refining and alignment module, including a speech large-model network, a segmentation large-model network, a visual feature refining module and a scene understanding module, inputting the sound signal and the visual signal in the multimodal information into the large-model-based visual-text refining and alignment module, and training the visual feature refining module based on a preset cross entropy loss, and training the scene module using a preset mean square error loss, to obtain the final extracted visual features and the final predicted text features, and to achieve preliminary alignment of the scene features and the text features;

[0011] S400: A feature extractor is used to convert the final extracted visual features and the final predicted text features into the same dimension of features, and a designed hash interaction robot skill learning module is used to perform feature interaction, achieve potential alignment of different modal features, and perform feature extraction, thereby predicting the corresponding robot action;

[0012] S500: Train the hash interactive robot skill learning module based on the teaching data and the preset loss function to obtain a trained hash interactive robot skill learning module, which is used to predict the robot's actions based on real-time visual and text input to complete the experimental task.

[0013] Preferably, S300 includes:

[0014] S310: The preset large speech model network extracts the sound signal and outputs the corresponding text information as text features ;

[0015] S320: and convert text features The prompt word and the RGB image of the current scene are input into the trained visual model for segmentation to obtain the visual features of the segmentation. , in Represents the number of image features extracted per unit time, Respectively represent the length and width of the captured data;

[0016] S330: The visual features obtained by the initial segmentation After being input into the visual feature refinement module, the final extracted visual features are output , the final extracted visual features Perform cross entropy loss with the segmentation labels of the actual collected operation scenes to train the visual feature refinement module;

[0017] S340: The final extracted visual features Input to the scene understanding module to predict the name of the segmented object to obtain the final predicted text features , in Represents the number of text features extracted per unit time, D represents the dimension of the final predicted text features, and the scene understanding module is trained based on the preset mean square error loss to achieve preliminary alignment of scene features and text features.

[0018] Preferably, in S400, a feature extractor is used to convert the final extracted visual features and the final predicted text features into the same dimension of features, specifically:

[0019] ;

[0020] ;

[0021] in, (.), (.) represent feature extractors, and Represent the final extracted visual features and the final predicted text features respectively, and Respectively represent visual features and text features of the same dimension.

[0022] Preferably, in S400, the designed hash interaction skill learning module is used to perform feature interaction, realize potential alignment of features of different modalities, and perform feature extraction, including:

[0023] S410: Visual features and text features Design and integrate adaptive hash features Ha;

[0024] S420: Calculate each visual feature and text features The cosine similarity of the fused adaptive hash vector Ha is used to obtain the cosine similarity between the visual feature and the hash vector and the cosine similarity between the text feature and the hash vector.

[0025] S430: Visual features and text features According to the corresponding visual feature cosine similarity and text feature cosine similarity sequence, they are sorted from large to small to obtain the sorted as well as And group n groups, each dimension of the visual features is , each dimension in the text feature is ;

[0026] S440: Concatenate the grouped features in the 0th dimension according to the grouping sequence number to obtain n concatenated features with a dimension of ( , M), perform attention learning on the spliced ​​features to complete feature alignment and splicing;

[0027] S450: Map the concatenated features into three vectors: query, key, and value. Calculate the attention weight and apply it to the value vector. After completing the lightweight hash self-attention mechanism, obtain the feature output after attention processing.

[0028] S460: These output features are fused and used for the final joint angle prediction.

[0029] Preferably, S410 specifically includes:

[0030] ;

[0031] in, The two features are concatenated in the 0th dimension, where the concatenated feature size is , It means taking the mean of the features in the 0th dimension, where the feature size after taking the mean is , Map the mean features to a learnable weight matrix, The dimension is , is a scalar bias term with a dimension of (1, M). The dimension of the fused adaptive hash feature obtained after this step is ;

[0032] S420 is specifically:

[0033] ;

[0034] ;

[0035] is the cosine similarity between the visual feature and the hash vector, is the cosine similarity between the text feature and the hash vector, is the formula for calculating cosine similarity, which is to perform dot product on the vector in the 0th dimension. and The dimensions are , ;

[0036] S450 is specifically:

[0037] ;

[0038] ;

[0039] ;

[0040] in, , , is a learnable weight matrix with dimension (M, M), D is the hidden dimension of the attention mechanism, , , is the corresponding bias vector with dimension (1, D). After this step, , , The dimensions of , M);

[0041] Calculate the attention weights and apply them to the value vector to get the feature output after attention processing:

[0042] ;

[0043] ;

[0044] Among them, softmax ( ) function is applied row by row, sqrt(D) represents the value of D under the square root, which is used to scale the dot product to prevent the gradient from disappearing. The dimension is .

[0045] Preferably, S460 includes:

[0046] S461: concatenate the feature outputs processed by attention in the 0th dimension to form a comprehensive feature representation;

[0047] S462: Perform dimensionality reduction processing on the comprehensive features to obtain features after dimensionality reduction;

[0048] S463: Introduce long short-term memory network to capture the temporal dependency of features after dimensionality reduction;

[0049] S464: Use the fully connected layer to predict the motion positions of each joint of the robot and the opening and closing angles of the gripper.

[0050] Preferably, S461 is specifically:

[0051] ;

[0052] in, The dimension is ;

[0053] S462 is specifically:

[0054] ;

[0055] in, is a learnable weight matrix with dimension (M, R), where R is the feature dimension after dimensionality reduction, R < M. is the bias vector, with dimension (1, R), The final dimension is ;

[0056] S463 is as follows:

[0057] ;

[0058] in, is the hidden state of the LSTM network, is the cell state, Indicates the current moment, Indicates the last moment, At the initial moment, it is taken as 0, so that the input features of the previous and next moments are linked. In the formula Represents the features extracted at the current moment;

[0059] S464 is specifically:

[0060] ;

[0061] in It means taking the mean of the 0th dimension, that is, the features per unit time Compressed to dimension 1, is the output weight matrix, with dimension , is the bias vector, with dimension (1, ), Represents the prediction result of the robot motion at the current moment, and its dimension is , is the number of joints that need to be predicted, 3 represents the three-dimensional spatial coordinates of each joint, and 2 represents two dimensions, which represent the opening and closing angles of the two grippers of the robot.

[0062] Preferably, the loss function preset in S500 is specifically:

[0063] ;

[0064] in It is the real coordinates and opening and closing angles of the expected robot movement in simulation experiments or drag teaching.

[0065] The above-mentioned large-model feature fusion hash self-attention method for robot operations effectively solves the problems of insufficient skill learning ability, poor generalization ability, and difficulty in precise control of robot systems in complex operating environments in the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a flow chart of a large model feature fusion hash self-attention method for robot operation in one embodiment of the present invention;

[0067] Figure 2 It is a schematic diagram of a large model-based visual-textual refinement and alignment module in one embodiment of the present invention;

[0068] Figure 3 This is a schematic diagram of a robot skill learning method with internal interaction of hash adaptive features in one embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0070] In one embodiment, Figure 1 As shown, a large model feature fusion hash self-attention method for robot operation, the method comprises the following steps:

[0071] S100: Acquire teaching data of various complex operation tasks in the target operation environment; further, collect teaching data of various common basic operations including but not limited to compliant grasping, large component joint handling, precision component assembly, etc., i.e., labeled data, corresponding to the real coordinates of the expected robot movement in the simulation experiment or drag teaching, through manual teaching or remote operation.

[0072] S200: Collects and integrates visual, acoustic and textual multi-dimensional information to fully capture multimodal information in the robot operation scene; further, uses high-speed industrial cameras and RGB-D cameras to obtain high-resolution images; uses microphone arrays to capture acoustic features during operation;

[0073] S300: constructing a large-model-based visual-text refining and alignment module, including a speech large-model network, a segmentation large-model network, a visual feature refining module and a scene understanding module, inputting the sound signal and the visual signal in the multimodal information into the large-model-based visual-text refining and alignment module, obtaining the final extracted visual features and the final predicted text features, and training the visual feature refining module based on a preset cross entropy loss, and training the scene module using a preset mean square error loss, to achieve preliminary alignment of scene features and text features;

[0074] S400: A feature extractor is used to convert the final extracted visual features and the final predicted text features into the same dimension of features, and a designed hash interaction robot skill learning module is used to perform feature interaction, achieve potential alignment of different modal features, and perform feature extraction, thereby predicting the corresponding robot action;

[0075] S500: Train the hash interactive robot skill learning module based on the teaching data and the preset loss function to obtain a trained hash interactive robot skill learning module, which is used to predict the robot's actions based on real-time visual and text input to complete the experimental task.

[0076] In one embodiment, Figure 2 As shown, S300 includes:

[0077] S310: The preset large speech model network extracts the sound signal and outputs the corresponding text information as text features ; Furthermore, the preset speech model is a trained speech model, and the sound signal is extracted through the trained speech model, and the corresponding text information is output as the text feature For subsequent skill learning;

[0078] S320: and convert text features The prompt word and the RGB image of the current scene are input into the trained visual model for segmentation to obtain the visual features of the segmentation. , in Represents the number of image features extracted per unit time, Respectively represent the length and width of the captured data;

[0079] Furthermore, since the generally designed scenarios are oriented towards specific requirements, the segmented features need to be further refined, so a feature refinement module is designed;

[0080] S330: The visual features obtained by the initial segmentation After being input into the visual feature refinement module, the final extracted visual features are output , the final extracted visual features Perform cross entropy loss with the segmentation labels of the actual collected operation scenes to train the visual feature refinement module;

[0081] S340: The final extracted visual features Input to the scene understanding module to predict the name of the segmented object to obtain the final predicted text features , in Represents the number of text features extracted per unit time, D represents the dimension of the final predicted text features, and the scene understanding module is trained based on the preset mean square error loss to achieve preliminary alignment of scene features and text features.

[0082] Furthermore, the scene understanding module is a dimensionality reduction feature extractor, which is jointly trained with the subsequent hash interaction skill learning module. The trained feature extractor can also be used. The final extracted text features are obtained in the previous step. and visual features However, due to the different feature dimensions of the two, it is difficult to use them for feature understanding of subsequent robots. Therefore, a feature extractor is needed. (.), (.),Will and Convert to features of the same dimension and .

[0083] In one embodiment, in S400, a feature extractor is used to convert the final extracted visual features and the final predicted text features into features of the same dimension, specifically:

[0084] ;

[0085] ;

[0086] in, (.), (.) represent feature extractors, and Represent the final extracted visual features and the final predicted text features respectively, and Respectively represent visual features and text features of the same dimension.

[0087] In one embodiment, in S400, the designed hash interaction skill learning module is used to perform feature interaction, realize potential alignment of features of different modalities, and perform feature extraction, including:

[0088] S410: Fusion Adaptive Hash Vector Extraction: for Visual Features and text features Design fusion adaptive hash feature Ha; S410 is specifically:

[0089]

[0090] in, The two features are concatenated in the 0th dimension, where the concatenated feature size is , It means taking the mean of the features in the 0th dimension, where the feature size after taking the mean is , Map the mean features to a learnable weight matrix, The dimension is , is a scalar bias term with a dimension of (1, M). The dimension of the fused adaptive hash feature obtained after this step is ;

[0091] S420: Similarity calculation: Calculate each visual feature and text features The cosine similarity of the fused adaptive hash vector Ha is used to obtain the cosine similarity of the visual feature and the hash vector and the cosine similarity of the text feature and the hash vector; S420 is specifically as follows:

[0092] ;

[0093] ;

[0094] is the cosine similarity between the visual feature and the hash vector, is the cosine similarity between the text feature and the hash vector, is the formula for calculating cosine similarity, which is to perform dot product on the vector in the 0th dimension. and The dimensions are , ;

[0095] S430: Feature grouping: grouping visual features and text features According to the corresponding visual feature cosine similarity and text feature cosine similarity sequence, they are sorted from large to small to obtain the sorted as well as And group n groups, each dimension of the visual features is , each dimension in the text feature is In this embodiment, n groups are 20, that is is divided into, , ,

[0096] , …, , the dimension of each group is .same Divided into , , , …, , the dimension of each group is ;

[0097] S440: Feature alignment and splicing: Splice the grouped features in the 0th dimension according to the grouping sequence number to obtain n spliced ​​features with a dimension of ( , M), perform attention learning on the spliced ​​features to complete feature alignment and splicing; further, regard each spliced ​​feature as a bucket, and then perform attention learning on the features in the bucket, while reducing the amount of calculation and strengthening the similarity between the aligned features;

[0098] S450: Lightweight hash self-attention mechanism: The concatenated features are mapped into three vectors: query, key, and value. The attention weight is calculated and applied to the value vector. After the lightweight hash self-attention mechanism is completed, the feature output after attention processing is obtained. S450 is specifically as follows:

[0099] ;

[0100] ;

[0101] ;

[0102] in, , , is a learnable weight matrix with dimension (M, M), D is the hidden dimension of the attention mechanism, , , is the corresponding bias vector with dimension (1, D). After this step, , , The dimensions of , M);

[0103] Calculate the attention weights and apply them to the value vector to get the feature output after attention processing:

[0104] ;

[0105] ;

[0106] Among them, softmax ( ) function is applied row by row, sqrt(D) represents the value of D under the square root, which is used to scale the dot product to prevent the gradient from disappearing. The dimension is ;

[0107] After S450, the self-attention mechanism between the aligned text and visual features is implemented. Since the attention mechanism is only applied to the aligned features in the bucket, the amount of calculation is greatly reduced. ,Since its input and output dimensions are the same, multiple lightweight hash self-attention mechanisms can be used for extraction, and 3 times is taken in practical applications;

[0108] S460: Feature fusion and joint angle prediction: These output features are fused and used for the final joint angle prediction.

[0109] After completing the lightweight hash self-attention mechanism, 20 feature outputs after attention processing were obtained. ,Next, these features need to be fused and used for the final joint angle prediction.

[0110] In one embodiment, Figure 3 As shown, S460 includes:

[0111] S461: Feature concatenation: The feature outputs after attention processing are concatenated in the 0th dimension to form a comprehensive feature representation; S461 is specifically as follows:

[0112] ;

[0113] in, The dimension is ;

[0114] S462: Feature dimensionality reduction: In order to reduce the computational complexity and extract the most critical information, the comprehensive features are subjected to dimensionality reduction processing to obtain the features after dimensionality reduction; S462 is specifically as follows:

[0115] ;

[0116] in, is a learnable weight matrix with dimension (M, R), where R is the feature dimension after dimensionality reduction, R < M. is the bias vector, with dimension (1, R), The final dimension is ;

[0117] S463: Considering the sequential nature of robot operations, a long short-term memory network is introduced to capture the temporal dependency of features after dimensionality reduction; S463 is specifically as follows:

[0118] ;

[0119] in, is the hidden state of the LSTM network, is the cell state, Indicates the current moment, Indicates the last moment, At the initial moment, it is taken as 0, so that the input features of the previous and next moments are linked. In the formula Represents the features extracted at the current moment;

[0120] S464: Use the fully connected layer to predict the motion positions of each joint of the robot and the opening and closing angles of the gripper.

[0121] S464 is specifically:

[0122] ;

[0123] in It means taking the mean of the 0th dimension, that is, the features per unit time Compressed to dimension 1, is the output weight matrix, with dimension , is the bias vector, with dimension (1, ), Represents the prediction result of the robot motion at the current moment, and its dimension is , is the number of joints that need to be predicted, 3 represents the three-dimensional spatial coordinates of each joint, and 2 represents two dimensions, which represent the opening and closing angles of the two grippers of the robot.

[0124] In one embodiment, the loss function preset in S500 is specifically:

[0125] ;

[0126] in It is the real coordinates and opening and closing angles of the expected robot movement in simulation experiments or drag teaching.

[0127] In this example, the Adam optimizer is used to minimize the loss function, and the learning rate is adjusted by using exponential decay after each round of training:

[0128] ;

[0129] in, is the initial learning rate (such as 0.001), is the decay rate (e.g. 0.95), is the number of decay steps (e.g. 1000). In this way, the model learns the mapping from visual and text features to joint angles, enabling the robot to learn complex skill operations. After training, the model can predict the corresponding robot actions based on new visual and text inputs, and has a certain generalization ability to adapt to new experimental tasks.

[0130] The beneficial effects of the present invention compared to the prior art are as follows:

[0131] 1. Comprehensive environmental perception capability: The present invention achieves a deep fusion of visual, acoustic, and text information through a large-model-based visual-text refinement and alignment module. The existing technology hardly considers the conversion and alignment operations between the three modes. This technology cleverly achieves a deep fusion of visual, acoustic, and text information by inputting text features extracted from acoustic signals as prompt words into the visual large model, and realigning the obtained segmented visual features with text features. This innovative alignment method significantly improves the robot's ability to understand complex environments, enabling it to more accurately identify and interpret objects and events in the scene, providing a more reliable basis for subsequent decision-making and control.

[0132] 2. Efficient feature learning: This invention introduces an innovative hash interactive robot skill learning module. By designing and integrating adaptive hash features and calculating cosine similarity, and selecting sorting and grouping, this grouping mode more accurately realizes the alignment of different modal features. Finally, self-attention calculation is performed through the most relevant features of different modalities, realizing efficient interaction between visual features and text features. Compared with the existing technology, this method not only greatly reduces the computational complexity, but also maintains the semantic association between features.

[0133] 3. Strong adaptability: Compared with the prior art, this invention combines a large model trained on a wide range of data with a lightweight hash self-attention mechanism, which can better generalize to the mode of a specified training scenario while performing efficient feature extraction and interactive learning. This approach enables the system to quickly learn and generalize to new tasks from limited teaching data. In addition, the introduction of the long short-term memory network (LSTM) enables the system to capture and utilize temporal information, further enhancing the learning and adaptability to complex task sequences.

[0134] 4. Accurate control performance: The present invention directly maps from multimodal input to accurate joint control through end-to-end learning, as shown in S464. This method closely combines perception and control, reducing the error accumulation in the intermediate links. In particular, the prediction results include the three-dimensional spatial coordinates and the opening and closing angles of the grippers of each joint of the robot, achieving fine control of the robot's movements and greatly improving the execution accuracy of complex operation tasks.

[0135] In terms of actual production operations, the technical benefits proposed by the present invention are as follows:

[0136] The large model feature fusion hash self-attention method of the present invention has brought significant improvements to the robot system in complex operating environments. Taking precision electronic manufacturing as an example, the technology effectively integrates visual input and voice commands through the visual-text refinement and alignment module of S300, enabling the robot to accurately identify and operate micro components. In particular, the hash interactive robot skill learning module in S400 realizes efficient multimodal information integration, greatly improving the system's adaptability to environmental changes. For example, when processing precision components such as small lenses, the robot can dynamically adjust its operating strategy according to real-time instructions. The lightweight hash self-attention mechanism introduced by S450 significantly improves the computing efficiency, enabling the system to respond quickly to complex operating environments. These technological innovations enable robots to show higher flexibility and efficiency in precision assembly tasks, and also greatly shorten the adaptation cycle of new products. In general, the present invention has demonstrated obvious technical advantages in improving operating accuracy, production efficiency and system adaptability, providing strong support for the intelligent upgrading of high-end manufacturing.

[0137] The above is a detailed introduction to a large model feature fusion hash self-attention method for robot operations provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A large model feature fusion hashing self-attention method for robot operation, characterized in that: The method comprises the following steps: S100: Acquire teaching data of various complex operation tasks in the target operation environment; S200: Collects and integrates visual, acoustic and textual multi-dimensional information to fully capture multi-modal information in the robot operation scene; S300: constructing a large-model-based visual-text refining and alignment module, including a speech large-model network, a segmentation large-model network, a visual feature refining module and a scene understanding module, inputting the sound signal and the visual signal in the multimodal information into the large-model-based visual-text refining and alignment module, obtaining the final extracted visual features and the final predicted text features, and training the visual feature refining module based on a preset cross entropy loss, and training the scene module using a preset mean square error loss, to achieve preliminary alignment of scene features and text features; S400: A feature extractor is used to convert the final extracted visual features and the final predicted text features into the same dimension of features, and a designed hash interaction robot skill learning module is used to perform feature interaction, achieve potential alignment of different modal features, and perform feature extraction, thereby predicting the corresponding robot action; S500: training the skill learning module of the hash interactive robot based on the teaching data and the preset loss function to obtain a trained skill learning module of the hash interactive robot, which is used to predict the robot's actions according to real-time visual and text inputs to complete the experimental task; S300 includes: S310: The preset large speech model network extracts the sound signal and outputs the corresponding text information as text features ; S320: and convert text features The prompt word and the RGB image of the current scene are input into the trained visual model for segmentation to obtain the visual features of the segmentation. , in Represents the number of image features extracted per unit time, Respectively represent the length and width of the captured data; S330: The visual features obtained by the initial segmentation After being input into the visual feature refinement module, the final extracted visual features are output , the final extracted visual features Perform cross entropy loss with the segmentation labels of the actual collected operation scenes to train the visual feature refinement module; S340: The final extracted visual features Input to the scene understanding module to predict the name of the segmented object to obtain the final predicted text features , in Represents the number of text features extracted per unit time, D represents the dimension of the final predicted text features, and the scene understanding module is trained based on the preset mean square error loss to achieve preliminary alignment of scene features and text features.

2. The method according to claim 1, characterized in that In S400, a feature extractor is used to convert the final extracted visual features and the final predicted text features into the same dimension of features, specifically: ； ； in, (.), (.) represent feature extractors, and Represent the final extracted visual features and the final predicted text features respectively, and Respectively represent visual features and text features of the same dimension.

3. The method according to claim 2, characterized in that S400 uses the designed hash interaction skill learning module to perform feature interaction, achieve potential alignment of different modal features, and perform feature extraction, including: S410: Visual features and text features Design and integrate adaptive hash features Ha; S420: Calculate each visual feature and text features The cosine similarity of the fused adaptive hash vector Ha is used to obtain the cosine similarity between the visual feature and the hash vector and the cosine similarity between the text feature and the hash vector. S430: Visual features and text features According to the corresponding visual feature cosine similarity and text feature cosine similarity sequence, they are sorted from large to small to obtain the sorted as well as And group n groups, each dimension of the visual features is , each dimension in the text feature is ; S440: Concatenate the grouped features in the 0th dimension according to the grouping sequence number to obtain n concatenated features with a dimension of ( , M), perform attention learning on the spliced ​​features to complete feature alignment and splicing; S450: Map the concatenated features into three vectors: query, key, and value. Calculate the attention weight and apply it to the value vector. After completing the lightweight hash self-attention mechanism, obtain the feature output after attention processing. S460: These output features are fused and used for the final joint angle prediction.

4. The method according to claim 3, characterized in that: S410 is specifically: ； in, The two features are concatenated in the 0th dimension, where the concatenated feature size is , It means taking the mean of the features in the 0th dimension, where the feature size after taking the mean is , Map the mean features to a learnable weight matrix, The dimension is , is a scalar bias term with a dimension of (1, M). The dimension of the fused adaptive hash feature obtained after this step is ; S420 is specifically: ； ； is the cosine similarity between the visual feature and the hash vector, is the cosine similarity between the text feature and the hash vector, is the formula for calculating cosine similarity, which is to dot product the vector in the 0th dimension and extract and The dimensions are , ; S450 is specifically: ； ； ； in, , , is a learnable weight matrix with dimension (M, M), D is the hidden dimension of the attention mechanism, , , is the corresponding bias vector with dimension (1, D). After this step, , , The dimensions of , M); Calculate the attention weights and apply them to the value vector to get the feature output after attention processing: ； ； Among them, softmax ( ) function is applied row by row, sqrt(D) represents the value of D under the square root, which is used to scale the dot product to prevent the gradient from disappearing. The dimension is .

5. The method according to claim 4, characterized in that S460 includes: S461: concatenate the feature outputs processed by attention in the 0th dimension to form a comprehensive feature representation; S462: Perform dimensionality reduction processing on the comprehensive features to obtain features after dimensionality reduction; S463: Introduce long short-term memory network to capture the temporal dependency of features after dimensionality reduction; S464: Use the fully connected layer to predict the motion positions of each joint of the robot and the opening and closing angles of the gripper.

6. The method according to claim 5, characterized in that S461 is as follows: ； in, The dimension is ; S462 is specifically: ； in, is a learnable weight matrix with dimension (M, R), where R is the feature dimension after dimensionality reduction, R < M. is the bias vector, with dimension (1, R), The final dimension is ; S463 is as follows: ； in, is the hidden state of the LSTM network, is the cell state, Indicates the current moment, Indicates the last moment, At the initial moment, it is taken as 0, so that the input features of the previous and next moments are linked. In the formula Represents the features extracted at the current moment; S464 is specifically: ； in It means taking the mean of the 0th dimension, that is, the features per unit time Compressed to dimension 1, is the output weight matrix, with dimension , is the bias vector, with dimension (1, ), Represents the prediction result of the robot motion at the current moment, and its dimension is , is the number of joints that need to be predicted, 3 represents the three-dimensional spatial coordinates of each joint, and 2 represents two dimensions, which represent the opening and closing angles of the two grippers of the robot.

7. The method according to claim 6, characterized in that The loss function preset in S500 is specifically: ； in It is the real coordinates and opening and closing angles of the expected robot movement in simulation experiments or drag teaching.

Images