Grabbing detection method based on cross-modal fusion and readable storage medium

Abstract

The invention relates to the technical field of image processing, and discloses a cross-modal fusion-based capture detection method and a readable storage medium, and the method comprises the steps: setting parallel RGB branches and depth branches in a coding stage, and carrying out the feature extraction of an RGB image and a depth image; in the RGB branches, a state space fusion module is added behind each residual block for feature fusion, and a fusion feature map with modal specificity and global relevance is generated; and the output of each down-sampling module in the RGB branch is connected to the corresponding up-sampling unit in the decoding stage in a jumping manner, and the position precision of grabbing detection and the accuracy of attitude estimation are further improved by multiplexing detail information of shallow layer features.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a grasping and detection method based on cross-modal fusion and a readable storage medium. Background Technology

[0002] Stable grasping of objects by grippers is a core capability for robot interaction with the environment, with broad application prospects in fields such as intelligent manufacturing, telemedicine, and aerospace. In practical applications, robots rely on visual perception to make grasping decisions, but due to the complexity of the environment and factors such as occlusion, robots face many problems in visual perception of the environment. Furthermore, the diversity of shapes, sizes, and postures of objects to be grasped poses a significant challenge to achieving reliable grasping in unstructured environments.

[0003] Traditional grasping and detection methods typically rely on prior geometric models of objects or accurate 3D reconstruction results, and then formulate specific grasping rules based on the known features of these models. However, in real-world scenarios, the objects to be grasped and the external environment are often unpredictable. When faced with unknown environments and unknown objects, the accuracy of grasping will be greatly affected, making it difficult to meet practical needs.

[0004] The increasing application of deep learning in the field of object detection has greatly promoted its development. Early object detection models used only RGB images as input. Although object detection can be considered a task similar to object detection, in practical deployments, the detection results are easily affected by environmental factors such as lighting and background. Therefore, subsequent research began to add depth maps to the input as supplementary spatial information. This raises the question of how the two modalities can cooperate effectively. RGB images contain color and texture information, while depth maps contain spatial geometric information. The effective utilization of complementary data from RGB and depth maps remains an unsolved challenge.

[0005] In terms of network architecture, convolutional neural networks (CNNs) have greatly promoted the development of grasping detection due to their ability to effectively extract local features and their translation invariance. However, the convolutional operation of CNNs limits their receptive field to a fixed kernel size, making it difficult to capture long-distance inter-pixel correlation information in images. This problem is particularly prominent in grasping detection tasks in complex scenes, and it also restricts further improvement in its accuracy.

[0006] In recent years, with the groundbreaking achievements of the Transformer architecture in natural language processing, researchers have introduced its core self-attention mechanism into computer vision, giving rise to the Visual Transformer (ViT) model. Unlike the local feature extraction of CNNs, ViT, through its global self-attention mechanism, can directly construct the dependency relationship between any two pixels in an image, extracting global contextual information and significantly improving its ability to capture long-distance associations. However, the global feature extraction capability brought by ViT's self-attention mechanism also has significant limitations. When processing high-resolution images commonly used in grasping and detection tasks, ViT incurs massive computational overhead and memory consumption, leading to a significant decrease in inference speed and making it difficult to meet the needs of grasping and detection in industrial scenarios. Furthermore, while ViT can model long-range dependencies, its shortcomings in processing local features when handling RGB-D bimodal data easily result in modal information imbalance and local detail dilution, hindering effective and accurate grasping.

[0007] In summary, existing cross-modal grasping and detection methods are difficult to adapt to the information specificity of the two modalities, and are prone to modal information imbalance and local detail dilution, resulting in reduced grasping accuracy. Summary of the Invention

[0008] Therefore, the technical problem to be solved by the present invention is to overcome the problems of modal information imbalance and local detail dilution that easily occur in the prior art, which leads to a decrease in grasping accuracy.

[0009] To address the aforementioned technical problems, this invention provides a grasping and detection method based on cross-modal fusion, comprising: The depth map of the target to be captured is obtained and input into the depth branch of the cross-modal fusion network. After passing through N residual blocks connected in sequence along the forward propagation direction, N depth feature maps of different scales are obtained. The RGB image of the target to be captured is obtained and input into the RGB branch of the cross-modal fusion network. After passing through N downsampling modules connected in series along the forward propagation direction, N fusion feature maps of different scales are obtained. The downsampling module includes a residual block and a state space fusion module connected in series along the forward propagation direction. The nth state space fusion module fuses the output of the RGB branch and the nth residual block in the depth branch to obtain a fusion feature map, which is used as the input of the (n+1)th residual block in the RGB branch, where 1≤n≤N. The depth feature map output by the Nth downsampling module in the RGB branch is passed through the (N+1)th residual block, and then through N upsampling units connected in series along the forward propagation direction. The upsampling feature map output by the last upsampling unit is obtained as the target refined feature map. The nth upsampling unit is skipped and connected to the nth state space fusion module. The refined feature map of the target is input into four parallel convolutional units to obtain the grasping quality map, angle cosine code map, angle sine code map and gripper width map of the target to be grasped, and then the grasping posture is generated to grasp the target.

[0010] Preferably, the nth state-space fusion module fuses the output of the RGB branch with the output of the nth residual block in the depth branch to obtain a fused feature map, including: The RGB feature map output by the nth residual block in the RGB branch is sequentially processed by global average pooling, 1×1 convolution, and 1×1 convolution, and then multiplied element-wise with the RGB feature map to obtain the enhanced RGB feature map. The depth feature map output by the nth residual block in the depth branch is sequentially processed by global average pooling, 1×1 convolution, and 1×1 convolution, and then multiplied element-wise with the depth feature map to obtain the enhanced depth feature map. After concatenating the RGB feature map and the depth feature map output by the nth residual block in the RGB branch and the depth branch along the channel dimension, batch normalization, non-linear mapping, 1×1 convolution and state space model are performed in sequence to obtain intermodal correlation features. The intermodal correlation feature map is multiplied element-wise with the enhanced RGB feature map and the enhanced depth feature map respectively, and then the elements are added together to obtain the fused feature map.

[0011] Preferably, the intermodal correlation features are expressed as: ; in, Indicates the intermodal correlation features, Representing a state-space model, Represents a 1×1 convolution. This represents a nonlinear mapping based on a multilayer perceptron. Indicates splicing along the channel. Represents the RGB feature map. Represents a depth feature map.

[0012] Preferably, after obtaining the fused feature map, the process includes: The fused feature map is sequentially passed through a 3×3 depthwise convolution, batch normalization, and ReLU activation function to obtain the optimized fused feature map.

[0013] Preferably, the upsampling unit is a lightweight feature refinement module, comprising: The upsampled feature map output by the (n+1)th lightweight feature refinement module and the fused feature map output by the nth downsampled module are used as the input of the nth lightweight feature refinement module. The input upsampled feature map is passed sequentially through a learned upsampled unit, a 3×3 convolutional unit, and a lightweight residual unit to obtain a residual feature map; The residual feature map is added element-wise to the input fused feature map to obtain the upsampled feature map; The learning-based upsampling unit includes an upsampling unit, a batch normalization layer, an activation function layer, a 3×3 depthwise convolutional unit, a batch normalization layer, and an activation function layer, which are connected in series along the forward propagation direction. The lightweight residual unit includes a 3×3 depth convolutional unit batch normalization layer, an activation function layer, and a 1×1 pointwise convolutional unit batch normalization layer and activation function layer connected in series along the forward propagation direction.

[0014] Preferably, generating a grasping posture for grasping the target includes: Obtain the location of the pixel with the largest pixel value in the capture quality image of the target to be captured, and use it as the target capture location; Based on the angle cosine encoding map and angle sine encoding map of the target to be grasped, the angle of the target is obtained at the target grasping position. The gripper width at the target gripping position in the gripper width diagram is taken as the target gripping width; The grasping pose is determined by the target grasping quality, target grasping angle, and target grasping width.

[0015] Preferably, after forming the grasping pose, the process includes: using a hand-eye conversion matrix to convert the grasping pose into a mechanical grasping posture in the mechanical world coordinate system, so that the robotic arm can grasp the target to be grasped.

[0016] Preferably, the angle cosine value at the target's grasping position is based on the angle cosine encoding map and the angle sine encoding map of the target to be grasped. With the sine of the angle Obtain the target capture angle , is represented as: .

[0017] Preferably, when generating the grasping posture to grasp the target, the process includes: constructing a loss function based on the grasping posture and the true label value, and optimizing the grasping posture; the loss function is expressed as: ; in, This represents the total number of pixels in the captured quality image. , and These represent the preset weighting coefficients for gripping quality, gripping angle, and gripping width, respectively. , and These represent the first three images in the grasp quality image, the angle sine code image, and the gripper width image, respectively. Predicted value at each pixel; , and These represent the label grabbing quality image, the label angle sine code image, and the label gripper width image, respectively. The true value of the label at each pixel.

[0018] This embodiment provides a readable storage medium on which a computer program is stored. When the computer program is executed, it implements the steps of the grasping and detection method based on cross-modal fusion as described above.

[0019] Compared with the prior art, the above-described technical solution of the present invention has the following advantages: The grasping detection method based on cross-modal fusion described in this invention sets up parallel RGB branches and depth branches in the encoding stage to extract features from RGB images and depth images respectively; in the RGB branch, a state space fusion module is added after each residual block to perform feature fusion, generating a fused feature map with modality specificity and global correlation; the output of each downsampling module in the RGB branch is connected to the corresponding upsampling unit in the decoding stage, and by reusing the detailed information of shallow features, the position accuracy and pose estimation accuracy of grasping detection are further improved.

[0020] The state-space fusion module of this invention adaptively learns the weights of each channel for depth and RGB features through an SE channel attention mechanism, increasing the weights of target texture and geometry channels while decreasing the weights of background and noise channels to enhance features. Simultaneously, it introduces a state-space model (SSM) to construct long-distance dependencies from texture to geometry with linear complexity, enriching the diversity of channel features and enabling cross-modal fusion. The state-space fusion module achieves intra-modal key feature enhancement and redundant information filtering, as well as long-distance dependency modeling with linear complexity between modalities. This highlights the advantages of each modality and the complementary fusion feature representation across modalities, generating fusion features with modality specificity and global relevance, ensuring the accuracy of subsequent grasping pose extraction.

[0021] This invention sets the upsampling unit in the decoder as a lightweight feature refinement module, and uses a learning-based upsampling unit to dynamically combine adjacent features, completing the progressive optimization of feature upsampling and detail enhancement, generating feature representations with both spatial resolution and discriminative power, effectively alleviating the information loss problem in the process of high-dimensional feature dimensionality reduction; at the same time, with the help of skip connections, it solves the problem that the simple interpolation upsampling commonly used in traditional decoders cannot learn feature dependencies, which not only ensures the consistency of global semantics, but also preserves the accuracy of local details, further improving the accuracy of grasping pose. Attached Figure Description

[0022] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 This is a schematic diagram of the grasping and detection method based on cross-modal fusion of the present invention. Figure 2 This is a structural diagram of the state-space fusion module; Figure 3 This is a structural diagram of the lightweight feature refinement module; Figure 4 This is a schematic diagram comparing the crawling and detection results of the present invention and different models on the Cornell dataset; Figure 5 This is a schematic diagram comparing the crawling and detection results of the present invention and different models on the Jacquard dataset. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0024] Reference Figure 1 As shown, the principle structure diagram of the grasping and detection method based on cross-modal fusion of the present invention specifically includes: The depth map of the target to be captured is obtained and input into the depth branch of the cross-modal fusion network. After passing through N residual blocks connected in sequence along the forward propagation direction, N depth feature maps of different scales are obtained. The RGB image of the target to be captured is obtained and input into the RGB branch of the cross-modal fusion network. After passing through N downsampling modules connected in series along the forward propagation direction, N fusion feature maps of different scales are obtained. The downsampling module includes a residual block and a state space fusion module connected in series along the forward propagation direction. The nth state space fusion module fuses the output of the RGB branch and the nth residual block in the depth branch to obtain a fusion feature map, which is used as the input of the (n+1)th residual block in the RGB branch, where 1≤n≤N. The depth feature map output by the Nth downsampling module in the RGB branch is passed through the (N+1)th residual block, and then through N upsampling units connected in series along the forward propagation direction. The upsampling feature map output by the last upsampling unit is obtained as the target refined feature map. The nth upsampling unit is skipped and connected to the nth state space fusion module. The refined feature map of the target is input into four parallel convolutional units to obtain the grasping quality map, angle cosine code map, angle sine code map and gripper width map of the target to be grasped, and then the grasping posture is generated to grasp the target.

[0025] Reference Figure 2 The diagram shows the structure of the state-space fusion module. The nth state-space fusion module fuses the output of the RGB branch with the output of the nth residual block in the depth branch to obtain a fused feature map, including: The RGB feature map output by the nth residual block in the RGB branch is sequentially processed by global average pooling, 1×1 convolution, and 1×1 convolution, and then multiplied element-wise with the RGB feature map to obtain the enhanced RGB feature map. The depth feature map output by the nth residual block in the depth branch is sequentially processed by global average pooling, 1×1 convolution, and 1×1 convolution, and then multiplied element-wise with the depth feature map to obtain the enhanced depth feature map. The RGB feature maps and depth feature maps output from the nth residual block in both the RGB and depth branches are concatenated along the channel dimension. Batch normalization, nonlinear mapping, 1×1 convolution, and a state-space model are then performed sequentially to obtain intermodal correlation features, represented as follows: ; in, Indicates the intermodal correlation features, Representing a state-space model, Represents a 1×1 convolution. This represents a nonlinear mapping based on a multilayer perceptron. Indicates splicing along the channel. Represents the RGB feature map. Represents a depth feature map; The intermodal correlation feature map is multiplied element-wise with the enhanced RGB feature map and the enhanced depth feature map respectively, and then the elements are added together to obtain the fused feature map.

[0026] In this embodiment, after obtaining the fused feature map, the fused feature map is sequentially passed through a 3×3 depthwise convolution, batch normalization, and ReLU activation function to obtain an optimized fused feature map.

[0027] The state-space fusion module of this invention adaptively learns the weights of each channel for depth and RGB features through an SE channel attention mechanism, increasing the weights of target texture and geometry channels while decreasing the weights of background and noise channels to enhance features. Simultaneously, it introduces a state-space model (SSM) to construct long-distance dependencies from texture to geometry with linear complexity, enriching the diversity of channel features and enabling cross-modal fusion. The state-space fusion module achieves intra-modal key feature enhancement and redundant information filtering, as well as long-distance dependency modeling with linear complexity between modalities. This highlights the advantages of each modality and the complementary fusion feature representation across modalities, generating fusion features with modality specificity and global relevance, ensuring the accuracy of subsequent grasping pose extraction.

[0028] Reference Figure 3 The diagram shown is a structural schematic of the lightweight feature refinement module; the upsampling unit in this embodiment is a lightweight feature refinement module, including: The upsampled feature map output by the (n+1)th lightweight feature refinement module and the fused feature map output by the nth downsampled module are used as the input of the nth lightweight feature refinement module. The input upsampled feature map is passed sequentially through a learned upsampled unit, a 3×3 convolutional unit, and a lightweight residual unit to obtain a residual feature map; The residual feature map is added element-wise to the input fused feature map to obtain the upsampled feature map; The learning-based upsampling unit includes an upsampling unit, a batch normalization layer, an activation function layer, a 3×3 depthwise convolutional unit, a batch normalization layer, and an activation function layer, which are connected in series along the forward propagation direction. The lightweight residual unit includes a 3×3 depth convolutional unit batch normalization layer, an activation function layer, and a 1×1 pointwise convolutional unit batch normalization layer and activation function layer connected in series along the forward propagation direction.

[0029] This invention sets the upsampling unit in the decoder as a lightweight feature refinement module, and uses a learning-based upsampling unit to dynamically combine adjacent features, completing the progressive optimization of feature upsampling and detail enhancement, generating a feature representation that has both spatial resolution and discriminative power. At the same time, with the help of skip connections, it solves the problem that the simple interpolation upsampling commonly used in traditional decoders cannot learn feature dependencies, which not only ensures the consistency of global semantics, but also preserves the accuracy of local details, further improving the accuracy of grasping pose.

[0030] Based on the above embodiments, in this embodiment of the invention, the crawling and detection method based on cross-modal fusion provided by the present invention is used to construct a crawling and detection network consisting of an encoder, a decoder, and an output head, and crawling and detection are performed, as shown in steps S201 to S204.

[0031] S201: Collect the RGB image and depth image of the target to be captured; The model input consists of an RGB image (3×H×W) and a depth image (1×H×W), which are preprocessed and uniformly adjusted to a size of 224×224 before being used as input.

[0032] S202: The encoder includes: constructing both the RGB image and depth image branches into a structure containing a first ResNet module, a second ResNet module, a third ResNet module, and a fourth ResNet module in series; inserting MSS modules between the ResNet modules of the RGB image branch, and using the outputs of the ResNet blocks of the RGB image and depth image at the same level as the inputs of the MSS.

[0033] Specifically, such as Figure 1 As shown, the encoder adopts a dual-branch parallel structure of RGB and depth maps. Both branches are composed of a first ResNet module, a second ResNet module, a third ResNet module, and a fourth ResNet module connected in series, used to extract features from shallow to deep layers step by step. Adjacent ResNet modules in the RGB branch are connected by an MSS (State Space Fusion) module. This module takes the output features of the RGB branch ResNet block and the corresponding ResNet block in the depth branch as input, performs accurate fusion of cross-modal features, and then inputs them to the next level ResNet block in the RGB branch. As the network depth increases, the feature map size is gradually reduced through max pooling, while the number of channels increases sequentially, forming a multi-scale feature sequence with feature sizes of 224×224×64, 112×112×128, 56×56×256, 28×28×512, and 14×14×1024, providing multi-level features for decoder fusion.

[0034] The MSS module in the encoder sequentially performs three processes: single-modal attention refinement, inter-modal correlation modeling, and fusion feature calibration, generating fusion features with modality specificity and global correlation; the specific implementation is as follows: Considering the specific information representation characteristics of RGB features and depth features—RGB features carry semantic information such as target texture and color, while depth features highlight structural information such as spatial geometry and distance—channel attention calibration is performed on both modal features separately to avoid noise interference in the fusion operation within a single modality. For the input RGB features... with depth features The process sequentially executes global average pooling, channel compression, channel restoration, and weight generation for channel attention. Global average pooling aggregates global statistical information for each channel, eliminating spatial redundancy. Then, Conv1×1 convolution compresses the number of channels to C / 16, reducing computational overhead while enhancing non-linear relationships between channels. After ReLU activation, Conv1×1 convolution restores the original number of channels C. Finally, a sigmoid function generates attention weights for each channel, which are then element-wise multiplied with the original features to obtain features that enhance key information. and This highlights the core features of the capture task and provides high-quality features for accurate cross-modal fusion.

[0035] The enhancement process of the RGB feature map is represented as follows: ; ; .

[0036] The enhancement process of the deep feature map is represented as follows: ; ; .

[0037] in, This is a global average pooling operation; This is a convolution operation with a 1x1 kernel and C / 16 output channels; This is a convolution operation with a 1x1 kernel and C output channels; This is the Sigmoid function.

[0038] Next, the original bimodal features and Concatenate along the channel dimension, eliminate intermodal numerical differences using BatchNorm, and then implement non-linear mapping of the feature dimension through MLP to enhance the semantic correlation of bimodal features. Next, input Conv1×1 to compress the number of feature channels from 2C to C. Finally, input the processed features into the State Space Model (SSM) to efficiently capture the long-range dependencies of bimodal features globally, solving the long-range modeling problem caused by the limitations of the local receptive field of convolution, and generating intermodal correlation features that contain complementarity between RGB and depth maps. , is represented as: .

[0039] intermodal correlation features Compared with the purified single-modal features , Dot product is used to integrate related information, and then element-wise addition is used to obtain preliminary fusion features. To further filter out redundancy and enhance feature consistency, The DWConv3×3 (depthmial convolution), BatchNorm, and ReLU activation functions are input sequentially to complete feature calibration and output the final fused features. , is represented as: ; .

[0040] The proposed multimodal fusion-based grasping detection method based on the SSM architecture can more accurately predict grasping quality and retain more detailed shape information. While Transformer-based grasping models capture global information through self-attention, and can model long-range dependencies, they are prone to modal information imbalance and local detail dilution when processing RGB-D bimodal data. Transformer's global modeling weakens key grasping details such as RGB texture edges and depth geometry contours, and the fusion of bimodal features is often a simple post-processing of concatenation, making it difficult to adapt to the information specificity of the two modalities. This invention, however, employs a two-stage design: single-modal attention purification using the MSS module and global correlation modeling using SSM. First, lightweight channel attention is used to enhance local details such as RGB target textures and depth spatial contours extracted by ResNet features. Then, SSM is used to capture long-range intermodal dependencies with linear complexity. This avoids the efficiency bottleneck of Transformer's self-attention computation while retaining the feature representation of both local and global information.

[0041] This invention performs cross-modal fusion by combining local feature extraction from convolutional neural networks with global modeling capabilities based on state-space models (SSM) that offer linear complexity. It also incorporates a lightweight feature refinement module to effectively mitigate the information loss problem during the dimensionality reduction of high-dimensional features. Furthermore, it reuses detailed information from shallow features to further improve the positional accuracy of grasping detection and the accuracy of pose estimation.

[0042] S203: The decoder includes: a fourth LFR module, a third LFR module, a second LFR module and a first LFR module connected in series; the MSS module and the LFR module at the same level are connected by skip connections; the output of the first LFR module is connected in four parallel Conv1x1 connections as the output.

[0043] Specifically, the decoder adopts a cascaded structure, consisting of a fourth LFR module, a third LFR module, a second LFR module, and a first LFR module connected sequentially. This structure progressively upsamples and refines the deep, low-resolution features output by the encoder, restoring the spatial detail information of the captured target. Feature supplementation is achieved between the encoder and decoder through skip connections. The output features of the MSS module are directly connected to the corresponding LFR modules in the decoder, providing rich shallow detail support for the decoding process.

[0044] The LFR module uses learned upsampling, channel fusion, and lightweight feature extraction as its core processes, achieving a dual improvement in feature resolution and discriminative power while ensuring lightweight operation. The specific implementation is as follows: First, the high-dimensional, low-resolution input features from the upper-layer module are... The learned upsampling units, namely DWConv3×3, BatchNorm, and ReLU, are sequentially input to increase the feature resolution to the target size, resulting in features rich in spatial information. Next, channel dimension transformation is performed through Conv3×3 convolution, while fusing cross-channel global semantic information; finally, lightweight residual units of DWConv3×3, BatchNorm, ReLU, PWConv1×1, BatchNorm, and ReLU are used to accurately extract local detail features while controlling computational cost, and finally, these features are combined with low-dimensional, high-resolution input features. The features are added together one by one, retaining their individual information, and the refined features are output. , is represented as: .

[0045] This invention constructs a learning-based upsampling operation, combined with a lightweight residual block, to form a lightweight feature refinement decoding module. This solves the problem that the simple difference upsampling commonly used in traditional decoders cannot learn feature dependencies, thus ensuring both the consistency of global semantics and the accuracy of local details.

[0046] S204: After the feature map is restored to the original input size by the decoder, it enters the output head to perform regression of grasping quality, angle and width. It outputs grasping quality map (Q), angle cosine code map (cos2θ), angle sine code map (sin2θ) and gripper width map (W) through four parallel 1x1 convolutional layers. The final grasping pose is input through these four heat maps.

[0047] The grasp representation is obtained from three heatmaps derived through regression: grasp quality, grasp angle, and grasp width. It is defined based on the grasp pose representation. ;in, This represents the capture quality of a specific pixel in the image, and its range is [0,1]. The higher the value, the better the pose. The higher the probability of successfully grabbing an object; This indicates the angle by which the end gripper rotates around the Z-axis of the object; This indicates the opening size of the end effector gripper. This gripping pose representation can effectively evaluate the quality score of each predicted grip, while simplifying the gripping task into a regression task on the predicted gripping position, angle, and width.

[0048] Capture Angle We obtain the following through calculation using sine and cosine: .

[0049] By capturing quality images The maximum value pixel in the range determines the grab position p, followed by Θ and The angle of this pixel is obtained from the image. The final grasping pose is determined by the opening size w. Finally, the grasping pose depicted in the image is converted into the corresponding grasping posture in the robot arm's world coordinate system using a hand-eye transformation matrix, and the robot performs the grasping task based on the result.

[0050] Based on the above embodiments, the present invention constructs the following loss function expression for the model based on the predicted grasping pose and the actual pose: ; in, It is the number of pixels in the image; , , It corresponds to the first of the three heat maps. The value of each pixel; , , Is it far from the first Extract the corresponding value of the rectangle label from the nearest dataset of pixels; , , These are the weighting coefficients for each part of the loss. Since the prediction of the center point and angle directly affects the detection accuracy, therefore... and Value setting ratio Larger, respectively 1.5, 1.5, and 1.

[0051] If the predicted capture box meets the angle and IOU constraints of this embodiment, it is considered a successful capture: the angle error between the predicted value and the label must be less than p / 6, and the IOU between the predicted value and the label must be greater than 25%.

[0052] In this invention, two public datasets, Cornell and Jacquard, are selected for training and testing the model. Each dataset is divided into two parts, with 90% used for training and 10% for testing. To measure the model's performance, detection accuracy is measured by both image (IW) and object (OW) partitioning. Image partitioning means randomly distributing the entire dataset in a 9:1 ratio to evaluate the model's ability to generalize to previously seen objects in different contexts and orientations. Object partitioning divides the dataset based on object instances to ensure that there are no completely identical object instances in the training and testing sets, thereby testing the model's ability to generalize to unknown objects.

[0053] Table 1 shows the performance comparison of the model with existing algorithms on the Cornell dataset. The accuracy in image and object tests reached 99.2% and 97.7% respectively, which is better than other algorithms in both metrics.

[0054] Table 1 Experimental results on the Cornell dataset Researchers method Data types IW OW Lenz SAE RGB-D 73.9 75.6 Chu ResNet-50 RGB-D 96 96.1 Kumra GR-CNN RGB-D 97.7 96.6 Wang TF-Grasp RGB-D 98 96.7 Zhang HCT-Grasp RGB-D 98.3 96.9 This invention Ours RGB-D 99.2 97.7 Table 2 shows the performance comparison of the model with existing algorithms on the Jacquard dataset; the method of this invention achieves a higher accuracy of 96.5%, which is higher than the previous methods.

[0055] Table 2 Experimental results on the Jacquard dataset Researchers method Data types IW OW Kumra GR-CNN RGB-D 92.6 87.7 Morrison TF-Grasp RGB-D 94.6 - Zhang HTC-Grasp RGB-D 95.8 92.4 Yu SKGNet RGB-D 95.9 - Chen SPANet RGB-D 96.1 - This invention Ours RGB-D 96.5 93.8 Results on the Cornell and Jacquard datasets show that the Transformer model and the hybrid grasping model have better accuracy on both datasets compared to traditional CNN models. The multimodal fusion grasping detection method based on the SSM architecture of this invention inherits the advantages of Transformer and CNN, and is more suitable for visual grasping tasks due to its linear complexity.

[0056] Reference Figure 4 The diagram shown is a comparison of the crawling and detection results of this invention and different models on the Cornell dataset; see reference. Figure 5 The diagram shown is a comparison of the crawling and detection results of the present invention and different models on the Jacquard dataset; Figure 4 and Figure 5It is known that the GR-CNN method, which is based solely on CNN, has low prediction quality in the central region of an object. This indicates that grasping pose detection is easily affected by environmental interference, which is due to the lack of an attention mechanism in the GR-CNN network, resulting in poor performance. Compared with the Transformer-based TF-Grasp model, the grasping detection method based on the SSM architecture proposed in this invention can predict grasping quality more accurately and retain more detailed shape information. This is because the TF-Grasp model relies on the Transformer's self-attention mechanism to capture global information. Although it can model long-range dependencies, it is prone to modal information imbalance and local detail dilution when processing RGB-D bimodal data. The global modeling of the Transformer weakens key grasping details such as RGB texture edges and depth geometric contours. Moreover, the fusion of bimodal features is mostly a simple post-processing of stitching, which is difficult to adapt to the information specificity of the two modalities. This invention employs a two-stage design: single-modal attention purification using the MSS module and global correlation modeling using SSM. First, it utilizes lightweight channel attention to enhance local details such as the target texture of RGB and the spatial contour of Depth. Then, it uses SSM to capture long-distance dependencies between modalities with linear complexity. This approach avoids the efficiency bottleneck of Transformer self-attention computation while preserving the feature representation of both local and global information.

[0057] In actual grasping experiments, a Jabot industrial robotic arm and a RealSense D435 RGB-D camera were used. The camera was fixed to the end effector to maintain good visual coverage of the graspable object. In each grasping attempt, the model received visual input from the depth camera on the robot's end gripper and output the optimal grasping pose. Then, the end gripper reached the optimal grasping position according to the trajectory planned by the motion planning method, and then the gripper was closed to attempt to grasp the object. The results show that the grasping detection method based on the SSM architecture also performs well on the real robotic arm.

[0058] The grasping detection method based on cross-modal fusion described in this invention sets up parallel RGB and depth branches in the encoding stage to extract features from RGB and depth images respectively. In the RGB branch, a state-space fusion module is added after each residual block to perform feature fusion, generating a fused feature map with modality specificity and global correlation. The output of each downsampling module in the RGB branch is connected to the corresponding upsampling unit in the decoding stage, further improving the position accuracy and pose estimation accuracy of grasping detection by reusing the detailed information of shallow features. The state-space fusion module of this invention adaptively learns the weights of each channel of depth and RGB features through the SE channel attention mechanism, increasing the weights of target texture and geometry channels and decreasing the weights of background and noise channels to achieve feature enhancement. At the same time, the state-space model SSM is introduced to construct long-distance dependencies from texture to geometry with linear complexity, enriching the diversity of channel features and realizing cross-modal fusion. The state-space fusion module enhances key features within a modality, filters redundant information, and models long-distance dependencies with linear complexity between modalities. It highlights the advantages of each modality and the complementary fusion feature representation across modalities, generating fusion features with modality specificity and global relevance, ensuring the accuracy of subsequent grasping pose extraction. This invention sets the upsampling unit in the decoder as a lightweight feature refinement module, using a learning-based upsampling unit to dynamically combine adjacent features, completing progressive optimization of feature upsampling and detail enhancement. This generates feature representations with both spatial resolution and discriminative power, effectively alleviating the information loss problem during high-dimensional feature dimensionality reduction. Simultaneously, combined with skip connections, it solves the problem that traditional decoders' simple interpolation upsampling cannot learn feature dependencies, ensuring both global semantic consistency and preserving the accuracy of local details, further improving the accuracy of grasping pose.

[0059] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0060] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0061] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0062] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0063] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A grasping and detection method based on cross-modal fusion, characterized in that, include: The depth map of the target to be captured is obtained and input into the depth branch of the cross-modal fusion network. After passing through N residual blocks connected in sequence along the forward propagation direction, N depth feature maps of different scales are obtained. The RGB image of the target to be captured is obtained and input into the RGB branch of the cross-modal fusion network. After passing through N downsampling modules connected in series along the forward propagation direction, N fusion feature maps of different scales are obtained. The downsampling module includes a residual block and a state space fusion module connected in series along the forward propagation direction. The nth state space fusion module fuses the output of the RGB branch and the nth residual block in the depth branch to obtain a fusion feature map, which is used as the input of the (n+1)th residual block in the RGB branch, where 1≤n≤N. The depth feature map output by the Nth downsampling module in the RGB branch is passed through the (N+1)th residual block, and then through N upsampling units connected in series along the forward propagation direction. The upsampling feature map output by the last upsampling unit is obtained as the target refined feature map. The nth upsampling unit is skipped and connected to the nth state space fusion module. The refined feature map of the target is input into four parallel convolutional units to obtain the grasping quality map, angle cosine code map, angle sine code map and gripper width map of the target to be grasped, and then the grasping posture is generated to grasp the target.

2. The grasping and detection method based on cross-modal fusion according to claim 1, characterized in that, The nth state-space fusion module fuses the output of the RGB branch with the output of the nth residual block in the depth branch to obtain a fused feature map, including: The RGB feature map output by the nth residual block in the RGB branch is sequentially processed by global average pooling, 1×1 convolution, and 1×1 convolution, and then multiplied element-wise with the RGB feature map to obtain the enhanced RGB feature map. The depth feature map output by the nth residual block in the depth branch is sequentially processed by global average pooling, 1×1 convolution, and 1×1 convolution, and then multiplied element-wise with the depth feature map to obtain the enhanced depth feature map. After concatenating the RGB feature map and the depth feature map output by the nth residual block in the RGB branch and the depth branch along the channel dimension, batch normalization, non-linear mapping, 1×1 convolution and state space model are performed in sequence to obtain intermodal correlation features. The intermodal correlation feature map is multiplied element-wise with the enhanced RGB feature map and the enhanced depth feature map respectively, and then the elements are added together to obtain the fused feature map.

3. The grasping and detection method based on cross-modal fusion according to claim 2, characterized in that, Intermodal correlation features are represented as follows: ； in, Indicates the intermodal correlation features, Representing a state-space model, Represents a 1×1 convolution. This represents a nonlinear mapping based on a multilayer perceptron. Indicates splicing along the channel. Represents the RGB feature map. Represents a depth feature map.

4. The grasping and detection method based on cross-modal fusion according to claim 2, characterized in that, After obtaining the fused feature map, the following is included: The fused feature map is sequentially passed through a 3×3 depthwise convolution, batch normalization, and ReLU activation function to obtain the optimized fused feature map.

5. The grasping and detection method based on cross-modal fusion according to claim 1, characterized in that, The upsampling unit is a lightweight feature refinement module, including: The upsampled feature map output by the (n+1)th lightweight feature refinement module and the fused feature map output by the nth downsampled module are used as the input of the nth lightweight feature refinement module. The input upsampled feature map is passed sequentially through a learned upsampled unit, a 3×3 convolutional unit, and a lightweight residual unit to obtain a residual feature map; The residual feature map is added element-wise to the input fused feature map to obtain the upsampled feature map; The learning-based upsampling unit includes an upsampling unit, a batch normalization layer, an activation function layer, a 3×3 depthwise convolutional unit, a batch normalization layer, and an activation function layer, which are connected in series along the forward propagation direction. The lightweight residual unit includes a 3×3 depth convolutional unit batch normalization layer, an activation function layer, and a 1×1 pointwise convolutional unit batch normalization layer and activation function layer connected in series along the forward propagation direction.

6. The grasping and detection method based on cross-modal fusion according to claim 1, characterized in that, Generate a grasping posture to grasp the target, including: Obtain the location of the pixel with the largest pixel value in the capture quality image of the target to be captured, and use it as the target capture location; Based on the angle cosine encoding map and angle sine encoding map of the target to be grasped, the angle of the target is obtained at the target grasping position. The gripper width at the target gripping position in the gripper width diagram is taken as the target gripping width; The grasping pose is determined by the target grasping quality, target grasping angle, and target grasping width.

7. The grasping and detection method based on cross-modal fusion according to claim 6, characterized in that, After forming the grasping pose, the process includes: using a hand-eye conversion matrix to convert the grasping pose into a mechanical grasping posture in the mechanical world coordinate system, so that the robotic arm can grasp the target to be grasped.

8. The grasping and detection method based on cross-modal fusion according to claim 6, characterized in that, Based on the angle cosine encoding map and angle sine encoding map of the target to be grasped, the angle cosine value at the target grasping position is... With the sine of the angle Obtain the target capture angle , is represented as: 。 9. The grasping and detection method based on cross-modal fusion according to claim 1, characterized in that, When generating a grasping posture to grasp a target, the process includes: constructing a loss function based on the grasping posture and the true label value, and optimizing the grasping posture; the loss function is expressed as: ； in, This represents the total number of pixels in the captured quality image. , and These represent the preset weighting coefficients for gripping quality, gripping angle, and gripping width, respectively. , and These represent the first three images in the grasp quality image, the angle sine code image, and the gripper width image, respectively. Predicted value at each pixel; , and These represent the label grabbing quality image, the label angle sine code image, and the label gripper width image, respectively. The true value of the label at each pixel.

10. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the steps of the grasping and detection method based on cross-modal fusion as described in any one of claims 1 to 9.

Images