A robotic arm grasping result detection method based on visual haptics and attention mechanism

Abstract

The application discloses a kind of based on visual tactile sensation and attention mechanism's mechanical arm grabbing result detection method, and grabbing result discriminant model according to visual-tactile sequence real-time data set, output article grabbing result information, the generation process of model includes: S1, the visual-tactile sequence initial data set of the article that mechanical arm grabs is acquired, data set is preprocessed and data is enhanced, and then it is divided into training set and test set;S2, training set is input into feature extraction module, obtain the space-time feature based on visual tactile sensation and input into two-stage fusion module, obtain fusion feature and input into grabbing result prediction module to train, obtain grabbing result discriminant initial model;S3, repeat step S2, obtain multiple grabbing result discriminant initial models and input test set, output the grabbing result discriminant intermediate model of highest accuracy as grabbing result discriminant model.Compared with prior art, the adaptability of grabbing model and the accuracy of grabbing result are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent robotic grasping technology, and in particular to a method for detecting the grasping results of a robotic arm based on visual-tactile and attention mechanisms. Background Technology

[0002] Industrial and service robots need to perform a range of grasping and complex manipulation tasks, and with the increasing demand for these robots, the field of robot manipulation has attracted widespread attention. Accurate and efficient grasping status assessment is a crucial component in improving the general grasping capabilities of robots.

[0003] Thanks to existing advancements in computer vision, computer vision-based environmental perception technologies have been extensively studied, with researchers achieving significant breakthroughs in areas such as grasping and slip detection, and material recognition. However, in vision-driven scenarios, due to numerous limitations in environmental perception, computer vision can only acquire limited information from a single perspective. This information primarily includes a complete scene description, which is often affected by exposure and focusing issues caused by changes in lighting and target materials. Due to these limitations, while pure vision techniques can provide solutions, their performance may still be insufficient in unstructured scenes. In situations where lighting conditions are far from ideal, or where unexpected obstacles obscure the target, or where complex interactions with the target are required, indirect measurement information such as vision may be inadequate. Problems such as complex backgrounds or missing operational details in the visual input often lead to task failure.

[0004] In contrast, tactile sensing information, obtained through direct contact with the object being manipulated, provides more localized and detailed information. It's also worth noting that different postures of the object during task execution can generate similar tactile signals, potentially negatively impacting the specific actions performed. Given the distinct characteristics of vision and touch, combining the strengths of both is a more effective approach.

[0005] However, in existing technologies, traditional CNN combined with LSTM methods for solving the visual-tactile information process struggles to extract detailed contact information and accurate pose information of the grasping object from the scene using CNN-based feature extraction. LSTM-based sequence feature extraction also struggles to effectively capture the relative relationships between multiple frames in a sequence, which is crucial for the continuity and stability of the entire extraction process. While methods incorporating attention mechanisms further consider intermodal correlations, they primarily apply attention to the input visual and tactile signal features, aiming for more effective feature extraction within a single modality. However, this approach neglects the relative importance of visual and tactile information across modalities. Furthermore, the extraction methods for spatiotemporal features from visual-tactile input series remain insufficient. Existing technologies only analyze depth images as the data source for spatial features, neglecting explicit consideration of the stability inherent in the temporal dimension and the grasping pose information inherent in the spatial dimension during the grasping process. The fusion of spatiotemporal features is rarely a focus of research. Summary of the Invention

[0006] The purpose of this invention is to overcome the problems in the prior art that it is difficult to obtain accurate pose information of the grasped object and ignore the relative importance of visual and tactile information between modalities, and to provide a robotic arm grasping result detection method based on visual-tactile and attention mechanisms.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0008] A method for detecting the grasping result of a robotic arm based on visual-tactile and attention mechanisms. The grasping result discrimination model outputs the grasping result information of the robotic arm based on a real-time dataset of visual-tactile sequences of objects grasped by the robotic arm. The generation process of the grasping result discrimination model specifically includes the following steps:

[0009] S1. Obtain the initial dataset of visual-tactile sequences of objects grasped by the robotic arm, preprocess and augment the initial dataset of visual-tactile sequences, and then split it into training set and test set;

[0010] S2. Input the training set into the feature extraction module to obtain spatiotemporal features based on visual information and spatiotemporal features based on tactile information, and input them into the two-stage fusion module to obtain visual-tactile fusion features. Input the visual-tactile fusion features into the grasping result prediction module for training to obtain the initial model for grasping result discrimination.

[0011] S3. Repeat step S2 to determine whether the number of training iterations has reached the training iteration threshold. If so, stop training to obtain multiple initial models for judging the crawling results. Input the test set into the initial models for judging the crawling results to obtain intermediate models for judging the crawling results. Use the intermediate model for judging the crawling results with the highest accuracy as the crawling result judging model.

[0012] As a preferred embodiment of the present invention, the preprocessing and data augmentation process in step S1 is specifically as follows: according to the preset data length of a single sample, the size of the visual image and the tactile image in the initial dataset of the visual-tactile sequence are adjusted respectively, the visual image and the tactile image are preprocessed respectively by subtracting images across frames, and the images are flipped according to a preset probability to perform data augmentation.

[0013] As a preferred embodiment of the present invention, in step S1, when performing data augmentation, the flipping method of the visual image and the tactile image is kept consistent for the same sample data. The processed visual-tactile sequence initial dataset is divided into training set, test set and validation set according to a certain ratio, and a data list and loading method are constructed for each dataset.

[0014] As a preferred embodiment of the present invention, the feature extraction module in step S2 is constructed based on an attention mechanism, including a temporal feature encoder and a spatial feature encoder.

[0015] As a preferred embodiment of the present invention, the time feature encoder extracts vision-based spatiotemporal features. Including temporal feature encoding from visual information and spatial feature encoding The spatial feature encoder extracts tactile-based spatiotemporal features, including temporal feature encoding from tactile information. and spatial feature encoding

[0016] As a preferred embodiment of the present invention, the image sequence pairs input to the initial visual-tactile sequence dataset are specifically represented as follows: and Where X v and X h These represent the input visual sequence and tactile sequence, respectively, with N, H, and W representing the sequence length, image height, and image width, respectively.

[0017] As a preferred embodiment of the present invention, the operation process of the spatial feature encoder is as follows:

[0018] S211. Take the last frame of the initial dataset of the visual-tactile sequence as the module input, and obtain the spatial blocks of a single frame image after the block operation.

[0019] S212. Encode each spatial block of a single frame image using a convolutional network and a fully connected layer to obtain the feature encoding of the single frame image.

[0020] S213. In the feature encoding of a single frame image, a spatial and channel-wise attention (SCA) mechanism is used to evaluate the importance of different sub-regions (corresponding to different blocks) of the feature encoding, and the scaled block features are obtained. The scaled block features are then projected by a multilayer perceptron to obtain a single-modal spatial feature encoding of visual and tactile information. The specific formula is shown below:

[0021]

[0022] Where F is the feature encoding, Pool is the pooling layer used in SCA, Sig is the Sigmod activation layer used to obtain importance scores for each part of the input, and MLP is a multilayer perceptron.

[0023] As a preferred embodiment of the present invention, the operation process of the time feature encoder is as follows:

[0024] S221. Extract images from each frame of the initial dataset of the visual-tactile sequence to obtain temporal blocks of a single frame image;

[0025] S222, The time blocks of the single-frame image are passed through convolutional layers and fully connected layers, and multiple single-frame images are combined to form multi-frame images. The multi-frame images are respectively converted into one-dimensional feature vectors.

[0026] S223. Calculate the corresponding query vector, key vector and value vector according to the one-dimensional feature vector, and apply a self-attention mechanism to the obtained vector to obtain the correlation information within the initial dataset of the visual-tactile sequence. After multiplying the correlation information with the original input information, obtain the temporal feature encoding of visual and tactile information through the mapping of a multilayer perceptron.

[0027] As a preferred embodiment of the present invention, the process of calculating the query vector, key vector, and value vector in step S223 specifically involves: the one-dimensional feature vector being compared with the independent query vector matrix W. Q , key vector matrix W K Sum value vector matrix W V Multiplying them yields the corresponding query vector Q, key vector K, and value vector V, as shown in the following formula:

[0028] Q = XW Q K = XW K V = XW V

[0029] Here, X is a one-dimensional feature vector, and the attention score QK for different locations in the image is obtained by calculating the dot product of the query vector Q and the key vector K. T After scaling the input, it is multiplied by the value vector V to obtain the attention score (Q, K, V) for single attention extraction. The attention score is then multiplied by the original input and mapped through a multilayer perceptron to obtain a single-modal temporal feature encoding.

[0030] As a preferred embodiment of the present invention, the fusion process of the dual-stage fusion module includes a cross-modal fusion stage and a cross-temporal fusion stage; temporal feature encoding from visual information. and spatial feature encoding Temporal feature encoding from tactile information and spatial feature encoding First, a preliminary fusion is performed through a cross-modal fusion stage, and then a cross-temporal fusion stage is performed to obtain visual-tactile fusion features.

[0031] As a preferred embodiment of the present invention, the cross-modal fusion stage of the dual-stage fusion module specifically includes the following steps:

[0032] S231. Input the spatiotemporal features based on visual information and the spatiotemporal features based on tactile information into the modal gating network to obtain the importance evaluation value for each feature. Multiply the importance evaluation value by the original spatiotemporal feature to obtain the corresponding weighted result feature. The specific formula is as follows:

[0033] F B =F⊙g(F)

[0034] Among them, F B To weigh the characteristics of the outcome, F is the feature encoding, and g(F) is the feature importance evaluation value;

[0035] S232. The obtained trade-off feature is concatenated and fused according to dimension, and then passed through a feedforward neural network to obtain preliminary temporal fusion features and preliminary spatial fusion features. The specific formula is as follows:

[0036]

[0037]

[0038] Among them, F Time For the initial fusion characteristics of time, F Space For the initial spatial fusion features, MLP stands for Multilayer Perceptron, and g represents the gating network acting on the modal features.

[0039] As a preferred embodiment of the present invention, the cross-temporal fusion stage of the dual-stage fusion module specifically includes the following steps:

[0040] S233. The preliminary temporal and spatial fusion features are passed through a spatiotemporal gating network, and then multiplied by themselves to obtain the scaled temporal and spatial features, respectively. The specific formulas are as follows:

[0041] F′ Time =F Time ⊙G(F Time )

[0042] F′ Space =F Space ⊙G(F Space )

[0043] Among them, F′ Time For the scaled time feature, F′ Space G represents the scaled spatial characteristics, and G is a spatiotemporal gating network.

[0044] S234. The scaled temporal and spatial features are concatenated, and a linear transformation operation is used to map the concatenated features to a low-dimensional feature space to obtain the visual-tactile fusion features, as shown in the following formula:

[0045] F fused =Concat(F′ Time F′ Space )·W T +b

[0046] Among them, F fused For visual-tactile fusion features, W T is the learnable parameter for performing the linear mapping in the model, and b is another set of learnable parameters used to provide the offset.

[0047] As a preferred embodiment of the present invention, the grasping result prediction module is constructed based on a fully connected neural network (FCN). By constructing a multilayer perceptron, the obtained visual-tactile fusion features are mapped to the two-dimensional space of decision-making to determine whether the grasping of the robotic arm in the current round is stable.

[0048] As a preferred embodiment of the present invention, the initial model for judging the capture result is provided with a loss function, specifically the cross-entropy loss function for binary classification tasks in machine learning, and a gradient descent algorithm is provided to optimize the loss function, the specific formula of which is shown below:

[0049]

[0050] Where Loss is the cross-entropy loss function, and y is the classification label. To classify the probability of belonging to label y, a gradient descent algorithm is used to optimize the loss function. After a complete round of calculation, the accuracy, precision, and recall of the initial model are evaluated on the test set to determine the crawling results.

[0051] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0052] 1. In the feature extraction stage, this invention clearly distinguishes between feature extraction of visual information and feature extraction of tactile information, and uses independent parameters to extract different characteristic features from visual and tactile information. At the same time, in the fusion method, this invention fully considers the importance of the continuity of the grasping process and the grasping posture. During fusion, a two-stage fusion method is adopted to adaptively adjust the importance of features from different sources, which effectively enhances the adaptability of the model and the accuracy of grasping result evaluation.

[0053] 2. In the feature extraction stage, this invention employs an attention mechanism, which can more effectively extract different features and their importance among different parts of a single frame image within the initial dataset of the visual-tactile sequence; and the attention visualization results show that the feature extraction method of this invention can effectively obtain the grasping position, the pose information of the target object, and the coherence within the input sequence.

[0054] 3. In the data processing stage, unlike other methods that use the difference between adjacent images, the present invention uses cross-frame subtraction for the input visual and tactile images, which increases the length of the window and can more prominently display the differences within the sequence in the feature preprocessing stage, effectively enhancing the feature extraction capability of the grasping result discrimination model in the time dimension.

[0055] 4. This invention employs a parallel feature extraction method for visual and tactile information inputs, simultaneously obtaining four-dimensional features from visual-tactile and temporal-spatial sources. Modal and spatiotemporal information are explicitly fused in stages through a two-step fusion approach. Furthermore, a gating network is introduced during the fusion stage to regulate the importance assessment of multiple inputs, thereby enhancing the model's adaptability. Attached image description:

[0056] Figure 1 This is a schematic flowchart of the detection method of the present invention;

[0057] Figure 2 This is a network structure diagram of the detection method of the present invention;

[0058] Figure 3 This is a schematic diagram of the data processing of the detection method of the present invention;

[0059] Figure 4 This is a schematic diagram of the attention distribution in spatial encoding of visual images in Embodiment 1 of the present invention;

[0060] Figure 5 This is a schematic diagram of the attention distribution in spatial encoding of tactile images in Embodiment 1 of the present invention;

[0061] Figure 6 This is a schematic diagram of the attentional relationship in the temporal encoding of the visual sequence in Embodiment 1 of the present invention;

[0062] Figure 7 This is a schematic diagram of the attentional relationship of the tactile sequence in time encoding in Embodiment 1 of the present invention;

[0063] Figure 8 This is a comparison chart of the accuracy evaluation results of the grasping state test in Embodiment 1 of the present invention. Detailed Implementation

[0064] The present invention will be further described in detail below with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0065] Unless otherwise specified, the use of terms such as "upper," "lower," "left," "right," "center," "inner," and "outer" to indicate orientation or positional relationships in the description of specific embodiments of the present invention is based on the orientation or positional relationships shown in the accompanying drawings, or the orientation or positional relationship in which the product / equipment / device is typically placed during use. These terms are merely for the purpose of facilitating the description of the present invention or simplifying the description in specific embodiments, enabling those skilled in the art to quickly understand the solution, and do not indicate or imply that a particular device / component / element must have a specific orientation, or be constructed and operated in a specific positional relationship. Therefore, they should not be construed as limitations on the present invention.

[0066] Furthermore, the use of terms such as "horizontal," "vertical," "suspended," and "parallel" does not imply that the corresponding device / component / element must be absolutely horizontal, vertical, suspended, or parallel, but rather that it can be slightly tilted or have a deviation. For example, "horizontal" merely means that its direction is more horizontal relative to "vertical," not that the structure must be completely horizontal, but that it can be slightly tilted. Alternatively, it can be simplified to mean that the corresponding device / component / element, when set in a "horizontal," "vertical," "suspended," or "parallel" direction, can have an error / deviation of ±10% relative to the corresponding direction, more preferably within ±8%, more preferably within ±6%, more preferably within ±5%, and more preferably within ±4%. As long as the corresponding device / component / element is within the error / deviation range, it can still achieve its function in the present invention.

[0067] Furthermore, the use of terms such as "first," "second," "third," etc. in terminology is merely for distinguishing identical or similar components and should not be interpreted as emphasizing or implying the relative importance of a particular component.

[0068] Furthermore, in the description of the embodiments of the present invention, "several", "more than", and "a number of" represent at least two. The number can be any number, such as 2, 3, 4, 5, 6, 7, 8, or 9, and can even exceed nine.

[0069] Furthermore, in the description of the technical solution of this invention, unless otherwise explicitly specified / limited / restricted, the terms "set up," "install," "connect," "link," "provided with," "laid out," and "arranged" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to common connection methods in the art, such as welding, riveting, bolting, and threaded connections. Such connections can be mechanical, electrical, or communication connections; they can be direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components.

[0070] Example 1

[0071] like Figure 1As shown, a method for detecting the grasping result of a robotic arm based on visual-tactile and attention mechanisms is presented. The grasping result discrimination model outputs the grasping result information of the robotic arm based on a real-time dataset of visual-tactile sequences of objects grasped by the robotic arm. In the feature extraction stage, different attention mechanisms are applied to extract features based on different information contained in the time and spatial dimensions, and these processes are performed synchronously in parallel to obtain the latent temporal and spatial features in the visual and tactile sequences. In the fusion stage, the complementarity and importance between visual-tactile information and between spatiotemporal information are fully considered, and a gating network is used for adjustment. This explicit, staged fusion further enhances the interpretability of the method. This invention effectively improves the accuracy of the model in predicting grasping results through a two-stage fusion approach of spatiotemporal parallel feature extraction and gating adjustment. The generation process of the grasping result discrimination model specifically includes the following steps:

[0072] S1. Obtain the initial dataset of visual-tactile sequences of objects grasped by the robotic arm. Preprocess and augment the initial dataset of visual-tactile sequences, and then split it into training set and test set. In this embodiment, the initial dataset of visual-tactile sequences is the slip dataset released by Jianhua et al., which contains 94 everyday items. During the collection process, different forces and positions are used to grasp the objects, and visual images and tactile information during the execution process are recorded.

[0073] S2. Input the training set into the feature extraction module to obtain the spatiotemporal features based on visual information and the spatiotemporal features based on tactile information, and input them into the two-stage fusion module to obtain the visual-tactile fusion features. Input the visual-tactile fusion features into the grasping result prediction module for training to obtain the initial model for grasping result discrimination.

[0074] S3. Repeat step S2 to determine whether the number of training iterations has reached the training iteration threshold. In this embodiment, the training iteration threshold is 200. If so, stop training and obtain multiple initial models for crawling result discrimination. Input the test set into the initial models for crawling result discrimination to obtain intermediate models for crawling result discrimination. Use the intermediate model for crawling result discrimination with the highest accuracy as the crawling result discrimination model.

[0075] The preprocessing and data augmentation process in step S1 is as follows: Based on the preset data length of a single sample, the sizes of the visual and tactile images in the initial dataset of the visual-tactile sequence are adjusted respectively. The visual and tactile images are preprocessed by subtracting images across frames respectively, and the images are flipped according to a preset probability to perform data augmentation. In specific implementation, for each capture execution round, that is, each capture record, eight consecutive frames of visual-tactile data are extracted as a set of input data, and the input data is uniformly adjusted to an image size of 224×224. The cross-frame subtraction method is used in the multi-frame input data to obtain images with more obvious changes.

[0076] like Figure 3 As shown, in step S1, when performing data augmentation, the flipping method of the visual image and the tactile image remains consistent for the same sample. In specific implementation, the preprocessed image is fed into the neural network. In this embodiment, a random flipping method is used to augment the data. For each sampled set of data, the vertical flipping probability and horizontal flipping probability of the current set of images are first determined randomly. Then, the same flipping operation is performed simultaneously on the visual image and the tactile image in this set. The processed visual-tactile sequence initial dataset is split into a training set, a test set, and a validation set according to a certain ratio, and a data list and loading method are constructed for each dataset. In specific implementation, the visual-tactile sequence initial dataset is divided into a training set, a test set, and a validation set in a 7:2:1 ratio. After the division is completed, the training set, the test set, and the validation set are stored in the form of npz files.

[0077] like Figure 2 As shown in this embodiment, the initial model for judging the crawling result includes a feature extraction module based on an attention mechanism, a two-stage fusion module based on a gating network to construct cross-modal fusion and cross-temporal fusion, and a crawling result prediction module based on a fully connected neural network.

[0078] In step S2, the feature extraction module is constructed based on an attention mechanism, including a temporal feature encoder and a spatial feature encoder. The temporal feature encoder extracts vision-based spatiotemporal features. Including temporal feature encoding from visual information and spatial feature encoding The spatial feature encoder extracts tactile-based spatiotemporal features, including temporal feature encoding from tactile information. and spatial feature encoding

[0079] The image sequence pairs input to the initial visual-tactile sequence dataset are specifically represented as follows: and Where X v and Xh These represent the input visual and tactile sequences, respectively. N, H, and W represent the sequence length, image height, and image width, respectively. In this embodiment, N is 8, and H... v W v H h W h Both are 224.

[0080] like Figure 4 and Figure 5 As shown, the operation process of the spatial feature encoder is as follows:

[0081] S211. Using the last frame of the initial dataset of the visual-tactile sequence as the module input, and performing a block-segmentation operation, spatial blocks of a single-frame image are obtained. In specific implementation, the visual input image X is used as the module input. v For example, after dividing it into blocks, spatial blocks of a single frame image are obtained. in It is the resolution of each block in the visual image. n is the number of blocks;

[0082] S212. Encode each spatial block of a single frame image using a convolutional network and a fully connected layer to obtain the feature encoding of the single frame image.

[0083] S213. In the feature encoding of a single frame image, a spatial and channel-wise attention (SCA) mechanism is used to evaluate the importance of different sub-regions (corresponding to different blocks) in the feature encoding, and obtain scaled block features. The scaled block features are then projected by a multilayer perceptron to obtain a single-modal spatial feature encoding of visual and tactile information. The specific formula is shown below:

[0084]

[0085] Where F is the feature encoding, Pool is the pooling layer used in SCA, Sig is the Sigmod activation layer used to obtain importance scores for each part of the input, and MLP is a multilayer perceptron.

[0086] In practice, only the last frame of the input sequence is used. First, it is arranged according to... Split into sub-graphs of fixed size X pIn this embodiment, the sub-image size is 28×28. The two-dimensional sub-image data is first processed through a small fully convolutional neural network to adjust the number of channels, and then a fully convolutional network is used to obtain sub-image features of a specified size. The positional encoding of the sub-image is constructed according to its position in the original image, and this encoding is added to the sub-image features to obtain the current image's encoding features F. The obtained image features are fed into the SCA attention mechanism to extract the importance of each block, and multiplied with the initial input to obtain the activation encoding features of the current image. Finally, through mapping by a multilayer perceptron, a single-modal spatial feature encoding is obtained.

[0087] like Figure 6 and Figure 7 As shown, the operation process of the time feature encoder is as follows:

[0088] S221. Extract images from each frame of the initial dataset of the visual-tactile sequence in blocks to obtain temporal blocks of a single frame image. Specifically, for a visual sequence, obtain... Where N is the length of the input sequence. This represents the resolution of each temporal block in a single frame image. n is the number of time blocks in each frame of the image;

[0089] S222: The temporal blocks of a single frame image are passed through convolutional and fully connected layers. Multiple single frame images are combined to form a multi-frame image. The multi-frame images are then converted into one-dimensional feature vectors.

[0090] S223. Calculate the corresponding query vector, key vector and value vector based on the one-dimensional feature vector, and apply a self-attention mechanism to the obtained vectors to obtain the correlation information within the initial dataset of the visual-tactile sequence. After multiplying the correlation information with the original input information, the temporal feature encoding of visual and tactile information is obtained through the mapping of the multilayer perceptron.

[0091] The process of calculating the query vector, key vector, and value vector in step S223 is as follows: the one-dimensional feature vector is respectively compared with the independent query vector matrix W. Q , key vector matrix W K Sum value vector matrix W V Multiplying them yields the corresponding query vector Q, key vector K, and value vector V, as shown in the following formula:

[0092] Q = XW Q K = XW K V = XW V

[0093] Here, X is a one-dimensional feature vector, and the attention score QK for different locations in the image is obtained by calculating the dot product of the query vector Q and the key vector K. TAfter scaling the input, it is multiplied by the value vector V to obtain the attention score (Q, K, V) for single attention extraction. The attention score is then multiplied by the original input and mapped through a multilayer perceptron to obtain a single-modal temporal feature encoding.

[0094] The fusion process of the two-stage fusion module includes a cross-modal fusion stage and a cross-temporal fusion stage; temporal feature encoding from visual information. and spatial feature encoding Temporal feature encoding from tactile information and spatial feature encoding First, a preliminary fusion is performed through a cross-modal fusion stage, and then a cross-temporal fusion stage is performed to obtain visual-tactile fusion features.

[0095] The cross-modal fusion stage of the two-stage fusion module specifically includes the following steps:

[0096] S231. Input the spatiotemporal features based on visual information and the spatiotemporal features based on tactile information into the modal gating network to obtain the importance evaluation value for each feature. Multiply the importance evaluation value by the original spatiotemporal feature to obtain the corresponding weighted result feature. The specific formula is as follows:

[0097] F B =F⊙g(F)

[0098] Among them, F B To weigh the characteristics of the outcome, F is the feature encoding, and g(F) is the feature importance evaluation value;

[0099] S232. The obtained trade-off feature is concatenated and fused according to dimension, and then passed through a feedforward neural network to obtain preliminary temporal fusion features and preliminary spatial fusion features. The specific formula is as follows:

[0100]

[0101]

[0102] Among them, F Time For the initial fusion characteristics of time, F Space For the initial spatial fusion features, MLP stands for Multilayer Perceptron, and g represents the gating network acting on the modal features.

[0103] The cross-temporal fusion stage of the two-stage fusion module specifically includes the following steps:

[0104] S233. The preliminary temporal and spatial fusion features are passed through a spatiotemporal gating network, and then multiplied by themselves to obtain the scaled temporal and spatial features, respectively. The specific formulas are as follows:

[0105] F′ Time =F Time ⊙G(F Time )

[0106] F′ Space =F Space ⊙G(F Space )

[0107] Among them, F′ Time For the scaled time feature, F′ Space G represents the scaled spatial characteristics, and G is a spatiotemporal gating network.

[0108] S234. The scaled temporal and spatial features are concatenated, and a linear transformation operation is used to map the concatenated features to a low-dimensional feature space to obtain the visual-tactile fusion features, as shown in the following formula:

[0109] F fused =Concat(F′ Time F′ Space )·W T +b

[0110] Among them, F fused For visual-tactile fusion features, W T is the learnable parameter for performing the linear mapping in the model, and b is another set of learnable parameters used to provide the offset.

[0111] The grasping result prediction module is built on a fully connected neural network (FCN). By constructing a multilayer perceptron, the obtained visual-tactile fusion features are mapped onto the two-dimensional decision space to determine whether the grasping of the robotic arm in the current round is stable. In this embodiment, the final feature output length of the grasping result prediction module is 2, which represents the probability of the grasping being stable and the probability of the grasping being unstable, respectively. In specific implementation, the expected output of the grasping result prediction module is 1 or 0, where 1 indicates a stable grasping and 0 indicates that the grasping has slipped.

[0112] The initial model for judging the captured results has a loss function, specifically the cross-entropy loss function used in binary classification tasks in machine learning. It also employs the Adam method to implement gradient descent, driving iterative optimization of the network parameters. The specific formula for the loss function is shown below:

[0113]

[0114] Where Loss is the cross-entropy loss function, and y is the classification label. To classify the probability of belonging to label y, a gradient descent algorithm is used to optimize the loss function. After a complete round of calculation, the accuracy, precision, and recall of the initial model are evaluated on the test set to determine the crawling results.

[0115] In this embodiment, TP (True Positives) represents true positive examples, which are predicted to be positive and are actually positive examples; FP (False Positives) represents false positive examples, which are predicted to be positive but are actually negative examples; FN (False Negatives) represents false negative examples, which are predicted to be negative but are actually positive examples; and TN (True Negatives) represents true negative examples, which are predicted to be negative and are actually negative examples.

[0116] The formula for calculating accuracy (Accc) is as follows:

[0117]

[0118] The formula for calculating accuracy Pre is as follows:

[0119]

[0120] The formula for calculating recall is as follows:

[0121]

[0122] like Figure 8 As shown, in this embodiment, the implementation effect of the algorithm of the present invention was analyzed and compared from three indicators: accuracy, precision, and recall. The experimental results and data obtained are presented. According to the established evaluation system and corresponding indicator requirements, compared with the prior art, the present invention has better stability and accuracy in detecting the grasping results of the robotic arm.

[0123] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting the grasping result of a robotic arm based on visual-tactile and attention mechanisms, characterized in that, The grasping result discrimination model outputs the grasping result information of the robotic arm based on the real-time dataset of the visual-tactile sequence of the object grasped by the robotic arm; The generation process of the crawling result discrimination model specifically includes the following steps: S1. Obtain the initial dataset of visual-tactile sequences of objects grasped by the robotic arm, preprocess and augment the initial dataset of visual-tactile sequences, and then split it into training set and test set; S2. Input the training set into the feature extraction module to obtain spatiotemporal features based on visual information and spatiotemporal features based on tactile information, and input them into the two-stage fusion module to obtain visual-tactile fusion features. Input the visual-tactile fusion features into the grasping result prediction module for training to obtain the initial model for grasping result discrimination. S3. Repeat step S2 to determine whether the number of training times has reached the training time threshold. If so, stop training and obtain multiple initial models for the crawling result discrimination. Input the test set into the initial models for the crawling result discrimination to obtain intermediate models for the crawling result discrimination. Use the intermediate model for the crawling result discrimination with the highest accuracy as the crawling result discrimination model. The fusion process of the dual-stage fusion module includes a cross-modal fusion stage and a cross-temporal fusion stage; the temporal and spatial features encoded from visual information and the temporal and spatial features encoded from tactile information are first initially fused through the cross-modal fusion stage, and then the visual-tactile fusion features are obtained through the cross-temporal fusion stage. The cross-modal fusion stage of the dual-stage fusion module specifically includes the following steps: S231. Input the spatiotemporal features based on visual information and the spatiotemporal features based on tactile information into the modal gating network to obtain the importance evaluation value for each feature. Multiply the importance evaluation value with the original spatiotemporal features to obtain the corresponding trade-off result feature. S232. The obtained trade-off result features are spliced ​​and fused according to dimensions and then passed through a feedforward neural network to obtain preliminary temporal fusion features and preliminary spatial fusion features. The cross-temporal fusion stage of the dual-stage fusion module specifically includes the following steps: S233. Pass the initial temporal fusion features and the initial spatial fusion features through a spatiotemporal gating network, and then multiply them by themselves to obtain the scaled temporal features and scaled spatial features. S234. The scaled temporal features and scaled spatial features are spliced ​​together, and the spliced ​​features are mapped to a low-dimensional feature space through a linear transformation operation to obtain visual-tactile fusion features.

2. The method for detecting the grasping result of a robotic arm based on visual-tactile and attention mechanisms according to claim 1, characterized in that, The preprocessing and data augmentation process in step S1 is as follows: Based on the preset data length of a single sample, the sizes of the visual and tactile images in the initial dataset of the visual-tactile sequence are adjusted respectively. The visual and tactile images are preprocessed by subtracting images across frames respectively, and the images are flipped according to a preset probability to perform data augmentation.

3. The method for detecting the grasping result of a robotic arm based on visual-tactile and attention mechanisms according to claim 1, characterized in that, In step S2, the feature extraction module is constructed based on the attention mechanism, including a temporal feature encoder and a spatial feature encoder.

4. The method for detecting the grasping result of a robotic arm based on visual-tactile and attention mechanisms according to claim 3, characterized in that, The time feature encoder and spatial feature encoder extract vision-based and tactile-based spatiotemporal features, including time feature encoding and spatial feature encoding from visual information, and time feature encoding and spatial feature encoding from tactile information.

5. The method for detecting the grasping result of a robotic arm based on visual-tactile and attention mechanisms according to claim 4, characterized in that, The operation process of the spatial feature encoder is as follows: S211. Take the last frame of the initial dataset of the visual-tactile sequence as the module input, and obtain the spatial blocks of a single frame image after the block operation. S212. Encode each spatial block of a single frame image using a convolutional network and a fully connected layer to obtain the feature encoding of the single frame image. S213. In the feature encoding of a single frame image, a spatial and channel attention mechanism is used to evaluate the importance of different sub-regions of feature encoding to obtain scaled block features. The scaled block features are then projected by a multilayer perceptron to obtain spatial feature encoding of visual and tactile information.

6. The method for detecting the grasping result of a robotic arm based on visual-tactile and attention mechanisms according to claim 4, characterized in that, The operation process of the time feature encoder is as follows: S221. Extract images from each frame of the initial dataset of the visual-tactile sequence to obtain temporal blocks of a single frame image; S222, The time blocks of the single-frame image are passed through convolutional layers and fully connected layers, and multiple single-frame images are combined to form multi-frame images. The multi-frame images are respectively converted into one-dimensional feature vectors. S223. Calculate the corresponding query vector, key vector and value vector according to the one-dimensional feature vector, and apply a self-attention mechanism to the obtained vector to obtain the correlation information within the initial dataset of the visual-tactile sequence. After multiplying the correlation information with the original input information, obtain the temporal feature encoding of visual and tactile information through the mapping of a multilayer perceptron.

7. The method for detecting the grasping result of a robotic arm based on visual-tactile and attention mechanisms according to claim 1, characterized in that, The initial model for judging the capture results has a loss function, which is specifically the cross-entropy loss function for binary classification tasks in machine learning.

Images