Grabbing detection method and device, computer device, storage medium and program product

Abstract

This invention relates to the field of image data processing technology, and discloses a grasping detection method, apparatus, computer equipment, storage medium, and program product. The method includes: acquiring a target image of a target to be grasped, the target image containing depth information and RGB information; processing the target image through a grasping detection model to obtain a grasping detection result; the grasping detection model includes a first Transformer module, a convolutional attention module, and a second Transformer module connected in sequence; calculating a grasping pose based on the grasping detection result, so that a robot can perform a grasping action on the target to be grasped according to the grasping pose. By setting the first Transformer module, convolutional attention module, and second Transformer module connected in sequence in the grasping detection model, the feature fusion capability of the grasping detection model can be significantly improved, and the accuracy of feature extraction can be increased.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, specifically to capture and detection methods, devices, computer equipment, storage media, and program products. Background Technology

[0002] Grasp detection refers to detecting the image of the object to be grasped to obtain the corresponding grasping pose of the robot's end effector gripper in the image, allowing the robot to grasp the object based on the grasping pose. However, low-light environments can affect the image, thus posing a challenge to grasp detection. How to perform grasp detection in low-light environments has become an urgent problem to be solved.

[0003] In related technologies, Transformer is introduced into the field of grasping detection to improve detection accuracy by using its ability to capture long-distance dependencies. This method usually embeds the Transformer module as an independent component into the encoder or decoder, aiming to introduce global modeling capabilities for different levels. However, this approach is not adaptable to different low-light environments, has weak robustness, and low detection accuracy and precision. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a grasping detection method, apparatus, computer equipment, storage medium and program product to solve the problem of low precision and accuracy in grasping detection in related technologies.

[0005] In a first aspect, the present invention provides a grasping detection method, the method comprising:

[0006] Acquire a target image of the target to be captured; the target image contains depth information and RGB information;

[0007] The target image is processed by a grasping detection model to obtain grasping detection results; the grasping detection model includes a first Transformer module, a convolutional attention module and a second Transformer module connected in sequence;

[0008] The grasping pose is calculated based on the grasping detection results, so that the robot can perform the grasping action on the target to be grasped according to the grasping pose.

[0009] In one optional implementation, the grasping detection model sequentially includes an encoder, a first Transformer module, a Transformer-CNN adapter, a convolutional attention module, a CNN-Transformer adapter, a second Transformer module, a feature fusion module, an upsampling module, and a decoder; the process of processing the target image through the grasping detection model to obtain the grasping detection result includes:

[0010] The target image is subjected to feature extraction by an encoder to obtain a first intermediate feature map;

[0011] The first intermediate feature map is processed by the first Transformer module using a window-based multi-head self-attention mechanism to obtain the second intermediate feature map;

[0012] The second intermediate feature map is adapted using a Transformer-CNN adapter to obtain the third intermediate feature map;

[0013] The third intermediate feature map is used to extract features through a convolutional attention module to obtain a fourth intermediate feature map;

[0014] The fourth intermediate feature map is adapted using a CNN-Transformer adapter to obtain the fifth intermediate feature map;

[0015] The fifth intermediate feature map is processed by the second Transformer module based on a multi-head self-attention mechanism to obtain the sixth intermediate feature map;

[0016] The feature fusion module performs feature fusion on the sixth intermediate feature map to obtain a feature fusion map;

[0017] The feature fusion map is upsampled using an upsampling module to obtain an upsampled feature map.

[0018] The upsampled feature map is decoded by a decoder to obtain the capture detection result.

[0019] In one optional implementation, the convolutional attention module includes a first branch and a second branch; the step of extracting features from the third intermediate feature map using the convolutional attention module to obtain a fourth intermediate feature map includes:

[0020] The third intermediate feature map is standardized by the first batch of normalization modules to obtain the first standardized feature map.

[0021] In the first branch, the first 1×k is passed sequentially. d Deep striped convolutional layer, first combined double normalization layer, first k d ×1 depth striped convolutional layer, second combined double normalization layer, first k c ×k c Convolutional layer, third combined double normalization layer and second k c ×k c The convolutional layer processes the first normalized feature map to obtain the first self-attention feature map;

[0022] In the second branch, the second k is passed sequentially. d ×1 depth striped convolutional layer, first combined double normalization layer, second 1×k d Deep striped convolutional layer, second combined double normalization layer, third k c ×k c Convolutional layer, third combined double normalization layer and fourth k c ×k c The convolutional layer processes the first normalized feature map to obtain the second self-attention feature map;

[0023] Add the first self-attention feature map and the second self-attention feature map to obtain the self-attention feature map;

[0024] The third intermediate feature map and the self-attention feature map are added together to obtain the summed feature map;

[0025] The additive feature map is processed sequentially through the second batch normalization module and the multilayer perceptron module to obtain the self-attention fusion feature map;

[0026] The summed feature map is added to the self-attention fusion feature map to obtain the fourth intermediate feature map.

[0027] In one optional implementation, acquiring the target image of the target to be captured includes:

[0028] The target camera acquires RGB and depth images of the target to be captured.

[0029] Convert the RGB image to a grayscale image and calculate the average brightness of the grayscale image;

[0030] Determine whether the average brightness of the grayscale image meets the low-light enhancement condition;

[0031] If the conditions are met, the RGB image is processed by the low-light enhancement module to obtain the enhanced RGB image;

[0032] The processed RGB image is added to the depth image to obtain the target image.

[0033] In one optional implementation, the low-light enhancement module includes a low-light enhancement student network and a low-light enhancement teacher network; the step of performing low-light enhancement processing on the RGB image through the low-light enhancement module to obtain an enhanced RGB image includes:

[0034] The RGB image is enhanced by a low-light enhancement student network using a low-light enhancement module to obtain an enhanced RGB image; wherein, the low-light enhancement teacher network updates the network weights of the low-light enhancement student network using a loss function corresponding to knowledge distillation.

[0035] In one optional implementation, the step of performing low-light enhancement processing on the RGB image through the low-light enhancement student network of the low-light enhancement module to obtain the enhanced RGB image includes:

[0036] The first feature map is obtained by extracting features from the RGB image through the first convolutional layer;

[0037] The second feature map is obtained by extracting features from the first feature map through the second convolutional layer;

[0038] The RGB image is element-wise multiplied with the second feature map to obtain the third feature map;

[0039] The first feature map is processed by the bias module to obtain the fourth feature map;

[0040] Add the first feature map to the fourth feature map to obtain the fifth feature map;

[0041] The sixth feature map is obtained by extracting features from the fifth feature map through the third convolutional layer;

[0042] The third feature map is added to the sixth feature map to obtain the seventh feature map;

[0043] The RGB image is used to extract features through the fourth convolutional layer to obtain the eighth feature map;

[0044] The eighth feature map is corrected using a correction module to obtain a color correction matrix and gamma parameters;

[0045] The color correction matrix is ​​processed by the first fully connected layer to obtain the ninth feature map;

[0046] The gamma parameters are processed by a second fully connected layer to obtain the target parameters;

[0047] Multiply the seventh feature map by the ninth feature map to obtain the tenth feature map;

[0048] The target parameters are exponentially combined with the tenth feature map to obtain the enhanced RGB image.

[0049] In a second aspect, the present invention provides a grasping detection device, the device comprising:

[0050] The image acquisition module is used to acquire a target image of the target to be captured; the target image contains depth information and RGB information.

[0051] The detection module is used to process the target image through a grasping detection model to obtain grasping detection results; the grasping detection model includes a first Transformer module, a convolutional attention module and a second Transformer module connected in sequence;

[0052] The pose calculation module is used to calculate the grasping pose based on the grasping detection result, so that the robot can perform the grasping action on the target to be grasped according to the grasping pose.

[0053] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the grasping and detection method of the first aspect or any corresponding embodiment described above.

[0054] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the grasping and detection method of the first aspect or any corresponding embodiment thereof.

[0055] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the capture and detection method of the first aspect or any corresponding embodiment described above.

[0056] The technical solution provided by this invention may include the following beneficial effects:

[0057] The grasping detection method provided by this invention first acquires a target image of the target to be grasped, which contains depth information and RGB information. Then, the target image is processed by a grasping detection model to obtain a grasping detection result. This grasping detection model includes a first Transformer module, a convolutional attention module, and a second Transformer module connected in sequence. Finally, the grasping pose is calculated based on the grasping detection result, so that the robot can perform the grasping action according to the grasping pose. By setting the first Transformer module, convolutional attention module, and second Transformer module connected in sequence in the grasping detection model, the advantages of global context modeling of Transformer and the local detail perception capability of convolution can be combined, significantly improving the feature fusion capability of the grasping detection model and improving the accuracy of feature extraction. Attached Figure Description

[0058] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0059] Figure 1 This is a flowchart illustrating the capture and detection method according to an embodiment of the present invention;

[0060] Figure 2 This is a flowchart illustrating another grasping and detection method according to an embodiment of the present invention;

[0061] Figure 3 This is a schematic diagram of the low-light enhancement module and the grasping detection network according to an embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram of the bias module according to an embodiment of the present invention;

[0063] Figure 5 This is a schematic diagram of the structure of the correction module according to an embodiment of the present invention;

[0064] Figure 6 This is a schematic diagram of the structure of the first Transformer module according to an embodiment of the present invention;

[0065] Figure 7 This is a schematic diagram of the structure of the second Transformer module according to an embodiment of the present invention;

[0066] Figure 8 This is a schematic diagram of the structure of a Transformer-CNN adapter according to an embodiment of the present invention;

[0067] Figure 9 This is a schematic diagram of the structure of a CNN-Transformer adapter according to an embodiment of the present invention;

[0068] Figure 10 This is a schematic diagram of the structure of the convolutional attention module according to an embodiment of the present invention;

[0069] Figure 11 This is a schematic diagram of the self-attention mechanism module according to an embodiment of the present invention;

[0070] Figure 12 This is a schematic diagram of the image involved in the capture and detection process according to an embodiment of the present invention;

[0071] Figure 13 This is a schematic diagram of the grasping pose according to an embodiment of the present invention;

[0072] Figure 14 This is a structural block diagram of a grasping and detection device according to an embodiment of the present invention;

[0073] Figure 15 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] Robotic grasping and inspection plays a crucial role in industry, improving operational efficiency and ensuring safety, especially in unstructured environments. Grasping and inspection refers to detecting the image of the object to be grasped to obtain the corresponding grasping pose of the robot's end effector, typically represented by a grasping rectangle. The robot can then grasp the object based on this pose. However, low-light environments can affect image quality, posing a challenge to grasping and inspection. Therefore, how to perform grasping and inspection in low-light conditions is a pressing issue that needs to be addressed.

[0076] In related technologies, multimodal sensing technology is used to integrate heterogeneous data sources such as infrared imaging, radar, and lidar to make up for the deficiencies of the visible spectrum. However, this approach is complex and costly.

[0077] According to an embodiment of the present invention, a method for grasping and detecting is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0078] This embodiment provides a crawling detection method that can be used on desktop computers, laptops, servers, etc. Figure 1 This is a flowchart of a capture detection method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0079] Step S101: Obtain the target image of the target to be captured.

[0080] The target to be grasped is the object that the robot needs to grasp. The target image contains depth information and RGB (red, green, and blue, the three primary colors of light) information. The target image can be obtained by taking pictures of the target with a camera with relevant functions, such as taking an RGB image of the target with a digital camera and taking a depth image of the target with a depth camera.

[0081] Step S102: Process the target image using the capture detection model to obtain the capture detection result.

[0082] This grasping detection model is used to perform grasping detection on target images. It detects the coordinates, angles, etc. required for the robot to perform grasping actions on the target as grasping detection results, so that the robot can perform grasping actions based on the grasping detection results.

[0083] The crawling detection model includes a first Transformer module, a convolutional attention module, and a second Transformer module connected in sequence, forming an alternating Transformer-convolution module. This combines the global context modeling advantages of Transformer with the local detail awareness capabilities of convolution, significantly improving the feature fusion capability of the crawling detection model and enhancing the accuracy of feature extraction.

[0084] Step S103: Calculate the grasping pose based on the grasping detection result, so that the robot can perform the grasping action on the target to be grasped according to the grasping pose.

[0085] After obtaining the grasping detection results, the grasping pose can be calculated based on the grasping detection results and the camera's extrinsic parameters. The robot can then perform grasping actions based on the grasping pose to grasp the target.

[0086] The grasping detection method provided in this embodiment first acquires a target image of the target to be grasped. This target image contains depth information and RGB information. Then, the target image is processed by a grasping detection model to obtain a grasping detection result. This grasping detection model includes a first Transformer module, a convolutional attention module, and a second Transformer module connected in sequence. Finally, the grasping pose is calculated based on the grasping detection result, so that the robot can perform the grasping action according to the grasping pose. By setting the first Transformer module, convolutional attention module, and second Transformer module connected in sequence in the grasping detection model, the advantages of global context modeling of Transformer and the local detail perception capability of convolution can be combined, significantly improving the feature fusion capability of the grasping detection model and increasing the accuracy of feature extraction.

[0087] This embodiment provides a crawling detection method that can be used on desktop computers, laptops, servers, etc. Figure 2 This is a flowchart of a capture detection method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0088] Step S201: Obtain the target image of the target to be captured.

[0089] The target image contains depth and RGB information.

[0090] Step S2011: Acquire RGB and depth images of the target to be captured using the target camera.

[0091] The target camera can be a color depth camera (RGBD camera), which has the ability to acquire RGB images and depth images simultaneously.

[0092] Step S2012: Convert the RGB image to a grayscale image and calculate the average brightness of the grayscale image.

[0093] Since low-light environments can affect the capture of RGB images, dim lighting can weaken the robustness of the grasping and detection algorithm and lead to feature degradation, the brightness of the RGB image can be judged, and low-light enhancement can be performed when the brightness is insufficient to facilitate subsequent feature extraction and improve the accuracy of grasping and detection.

[0094] Specifically, the RGB image is converted to grayscale to convert it from a color image to a grayscale image in order to calculate the average brightness of the grayscale image.

[0095] Step S2013: Determine whether the average brightness of the grayscale image meets the low-light enhancement condition.

[0096] After calculating the average brightness of the grayscale image, a lighting gating mechanism is set to automatically adjust the activation state of the low-light enhancement module. Specifically, the average brightness of the grayscale image is compared with a preset low-light enhancement condition to determine whether low-light enhancement of the RGB image is necessary. This low-light enhancement condition can be set according to actual needs, such as setting a brightness threshold. When the average brightness of the grayscale image is less than the brightness threshold, it is determined that the low-light enhancement condition is met. For example, the brightness threshold is 0.3.

[0097] Step S2014: If the conditions are met, the RGB image is processed by the low-light enhancement module to obtain the enhanced RGB image.

[0098] If the average brightness of the grayscale image meets the low-light enhancement conditions, it means that the RGB image was taken in a low-light environment and needs to be low-light enhanced. This can be done by using a low-light enhancement module to process the RGB image and obtain the enhanced RGB image.

[0099] Optionally, the low-light enhancement module includes a low-light enhancement student network and a low-light enhancement teacher network. The low-light enhancement teacher network is a neural network with knowledge of low-light enhancement or dark image enhancement, such as a dark image enhancement network based on the Transformer architecture. This low-light enhancement teacher network is pre-trained on a large-scale low-light enhancement dataset and possesses strong feature representation capabilities. By employing knowledge distillation techniques, the rich prior knowledge encapsulated in the low-light enhancement teacher network is transferred to the low-light enhancement student network. This avoids complex image preprocessing and fully utilizes high-quality visual information to guide model optimization. This knowledge transfer not only provides a better initialization foundation for the model but also stabilizes the training process and imposes stronger regularization constraints on the low-light enhancement learning network. By fully utilizing the rich prior knowledge of the teacher network, its adaptability in low-light environments is significantly improved. The low-light enhancement student network is used to model relevant parameters in the image signal processing (ISP) process, significantly reducing computational complexity while maintaining competitive performance. The low-light enhancement teacher network updates the network weights of the low-light enhancement student network through the loss function corresponding to knowledge distillation. This transfers low-light enhancement or dark-light enhancement knowledge to the more compact and lightweight low-light enhancement student network, ensuring the overall real-time performance and low parameter count of the crawling and detection system (including the low-light enhancement module and the crawling and detection model). When performing low-light enhancement processing on the RGB image through the low-light enhancement module, the low-light enhancement student network of the low-light enhancement module performs low-light enhancement processing on the RGB image to obtain the enhanced RGB image.

[0100] For example, Figure 3 This is a schematic diagram of the low-light enhancement module and the grasping detection network according to an embodiment of the present invention. Figure 3As shown, the low-light enhancement module includes branch A and branch B. Branch A is used to map the sRGB image to the original RGB space, and branch B is used to model white balance and color space conversion as well as perform gamma correction. When performing low-light enhancement processing on the RGB image through the low-light enhancement student network of the low-light enhancement module, in branch A, the first feature map is first extracted from the RGB image through the first convolutional layer (Conv). Then, the first feature map is extracted through the second convolutional layer (Conv) to obtain the second feature map. Then, the RGB image and the second feature map are multiplied element-wise to obtain the third feature map. This element-wise multiplication is used to indicate pixel-level matrix multiplication, that is, the elements at each corresponding position in two matrices of the same size are multiplied respectively. At the same time, the first feature map is processed through the bias module to obtain the fourth feature map. The bias module is connected to the first convolutional layer through a residual connection to provide a non-fading gradient propagation channel and alleviate the gradient vanishing in deep networks. Then, the first feature map and the fourth feature map are added to obtain the fifth feature map. This addition is used to indicate element-wise addition. Then, the fifth feature map is extracted through the third convolutional layer (Conv) to obtain the sixth feature map. The third feature map and the sixth feature map are added to obtain the seventh feature map. In branch B, the RGB image is feature-extracted through the fourth convolutional layer (Conv) to obtain the eighth feature map. This eighth feature map is then corrected by the Calibration Module to obtain the Color Correction Matrix and gamma parameters. The Color Correction Matrix is ​​then processed by the first fully connected layer (FC) to obtain the ninth feature map. The gamma parameters are then processed by the second fully connected layer (FC) to obtain the target parameters. Next, the seventh feature map obtained from branch A is multiplied by the ninth feature map obtained from branch B. This matrix multiplication indicates that the two matrices are being multiplied, resulting in the tenth feature map. Finally, the target parameters obtained from branch B are exponentially multiplied by the tenth feature map. Based on the target parameters, each element in the tenth feature map is exponentially multiplied to obtain the enhanced RGB image. For example, the color correction matrix is ​​a 3×3 matrix containing 9 color parameters.

[0101] The correction module optimizes image color and illumination distribution. The eighth feature map input from the fourth convolutional layer includes key parameters (K, used as an identifier) ​​and value parameters (V, indicating the original information). In this embodiment, a learnable query parameter (Q, used to find the required information; in this embodiment, the query parameter includes 10 parameters, corresponding to 9 parameters and 1 gamma parameter of the color correction matrix). The color correction matrix corrects white balance and color deviation, mapping the original color space to a standard color space. The gamma parameter maps linear light signals to a brightness space that conforms to human visual perception.

[0102] Optional, Figure 4 This is a schematic diagram of the bias module according to an embodiment of the present invention. In this bias module, the first feature map is processed sequentially by LightNorm (lightweight normalization) and 1×1 convolution (1×1 Conv), and then multiplied by parameter a1 to obtain the first bias feature map. The first feature map is added to the first bias feature map to obtain the second bias feature map. The second bias feature map is processed sequentially by LightNorm and 1×1 convolution, and then multiplied by parameter a2 to obtain the third bias feature map. The second bias feature map is added to the third bias feature map to obtain the fourth feature map. a1 and a2 represent two learnable small values, introduced to ensure convergence stability.

[0103] Optional, Figure 5 This is a schematic diagram of the structure of the correction module according to an embodiment of the present invention. In this correction module, the eighth feature map is processed by a residual module containing a 3×3 depthwise convolutional layer (3×3DWConv) to obtain the first corrected feature map. Then, the first corrected feature map is processed by a Transformer-based attention mechanism. Specifically, the first corrected feature map is processed by two fully connected layers (FC) to obtain the key parameter K and the value parameter V. The query parameter Q is multiplied by the key parameter K and then multiplied by the value parameter V. After that, it is processed by a fully convolutional layer (FC) again, and then processed by a residual module containing LayerNorm (layer normalization) and a fully connected layer to obtain the module output.

[0104] Step S2015: Add the processed RGB image to the depth image to obtain the target image.

[0105] After processing the RGB image using the low-light enhancement module, the processed RGB image is added to the depth image, and element-level feature overlay and fusion are performed to obtain the target image.

[0106] Optionally, the depth image and the processed RGB image can be processed separately using two 3×3 convolutional layers (Conv) and then added together to obtain the target image.

[0107] Step S2016: If the condition is not met, add the RGB image to the depth image to obtain the target image.

[0108] If the average brightness of the grayscale image does not meet the low-light enhancement conditions, it means that the RGB image was taken under normal lighting conditions. Therefore, there is no need to perform low-light enhancement processing on the RGB image. The RGB image can be directly added to the depth image to obtain the target image.

[0109] Optionally, the depth image and RGB image can be processed separately using two 3×3 convolutional layers and then added together to obtain the target image.

[0110] By setting up this illumination gating mechanism, the problem of performance degradation caused by feature space misalignment due to direct cascading of the low-light enhancement module and the grasping detection model is avoided. This enables collaborative optimization between the low-light enhancement module and the grasping detection model, thereby improving the accuracy of grasping detection.

[0111] Step S202: Process the target image using the capture detection model to obtain the capture detection result.

[0112] Optionally, the grasping and detection model includes a first Transformer module based on a window-based multi-head self-attention mechanism, a convolutional attention module, and a second Transformer module based on a multi-head self-attention mechanism, connected sequentially. The first Transformer module based on the window-based multi-head self-attention mechanism divides the feature map of the target image into multiple non-overlapping local windows, with self-attention calculated independently within each local window, significantly reducing computational complexity while maintaining global relationship modeling capabilities. The convolutional attention module utilizes convolution to simulate the basic structure of a Transformer, thus better adapting to other Transformer-based modules and improving the perception of local image details. The second Transformer module based on the multi-head self-attention mechanism extracts global contextual information of the image for modeling, processing features with reduced resolution after passing through the first Transformer module, improving feature extraction accuracy while reducing computational complexity. By setting up a first Transformer module, a convolutional attention module, and a second Transformer module based on a multi-head self-attention mechanism connected in sequence, the computational complexity can be significantly reduced while maintaining the ability to model global relationships, improving the ability to perceive local details of the image, preserving the global contextual information of the image, and improving the accuracy and efficiency of grasping and detection.

[0113] Optionally, the grasping and detection model sequentially includes an encoder, an alternating Transformer-CNN module, and a decoder. The alternating Transformer-CNN module comprises, in sequence, a first Transformer module, a Transformer-CNN adapter, a convolutional attention module, a CNN-Transformer adapter, a second Transformer module, a feature fusion module (DAPPM, Dense Atrous Spatial Pyramid Pooling Module), and an upsampling module. Since traditional CNN models have limited receptive fields, and pure Transformer architectures lack sufficient local feature modeling, the alternating Transformer-CNN module of this invention innovatively connects the encoder and decoder through a progressive fusion paradigm of alternating Transformer and CNN. Unlike shallow embedded fusion where Transformer modules are simply embedded as independent components in the encoder or decoder with CNNs connected in series or parallel, this constructs a collaborative framework where global context guides local optimization, and local optimization feeds back into global understanding. It achieves deep synergy between the two architectures rather than simple stacking, cleverly integrating the global context modeling advantages of Transformer with the local detail perception capabilities of CNN, realizing synergistic complementarity between global context modeling and local feature extraction. As a core component connecting the encoder and decoder, this alternating Transformer-CNN module provides a more powerful feature representation for the decoder by fusing and optimizing the multi-scale features extracted by the encoder. By employing cross-block adapters—namely, the Transformer-CNN adapter and the CNN-Transformer adapter—to connect the first Transformer module, the convolutional attention module, and the second Transformer module, the model's feature fusion capability is significantly improved.

[0114] Specifically, when processing the target image using the capture detection model, the encoder first extracts features from the target image to obtain a first intermediate feature map. Then, the first Transformer module processes this first intermediate feature map using a window-based multi-head self-attention mechanism. This divides the feature map into non-overlapping local windows, with self-attention calculated within each window. This reduces computational complexity while maintaining global relationship modeling capabilities, resulting in a second intermediate feature map. Next, the Transformer-CNN adapter adapts this second intermediate feature map to obtain a third intermediate feature map. A convolutional attention module then extracts features from this third intermediate feature map to obtain a fourth intermediate feature map. Finally, the CNN-Transformer adapter adapts this fourth intermediate feature map to obtain a fifth intermediate feature map. By setting up Transformer-CNN and CNN-Transformer adapters, progressive feature fusion is performed while preserving the inherent characteristics of each modality. Two cascaded deep striped convolutional layers are used to achieve near-standard large-kernel convolution operations, significantly improving computational efficiency while effectively capturing long-range contextual dependencies. The Transformer-CNN adapter and CNN-Transformer adapter also integrate standard convolutional layers for local feature interaction and cross-channel fusion, and combine them with average pooling layers to achieve spatial information aggregation. This combined structure enables efficient and semantically coherent feature transformation and alignment. Next, the fifth intermediate feature map is processed by a multi-head self-attention mechanism in the second Transformer module to handle the reduced-resolution features, resulting in a sixth intermediate feature map. Then, the feature fusion module performs multi-scale feature fusion on the sixth intermediate feature map to obtain a fused feature map. The upsampling module then upsamples this fused feature map to obtain an upsampled feature map. Finally, the decoder decodes this upsampled feature map to obtain the grasping detection result.

[0115] Optional, Figure 6 This is a schematic diagram of the structure of the first Transformer module according to an embodiment of the present invention. The first Transformer module includes, in sequence, a LayerNorm module, a W-MSA (Window-based Multi-head Self-Attention) module, a LayerNorm module, and an MLP (Multilayer Perceptron) module.

[0116] Optional, Figure 7 This is a schematic diagram of the structure of the second Transformer module according to an embodiment of the present invention. The second Transformer module includes, in sequence, a LayerNorm module, an MSA (Multi-head Self-Attention) module, a LayerNorm module, and an MLP module.

[0117] Optional, Figure 8 This is a schematic diagram of the Transformer-CNN adapter according to an embodiment of the present invention. The Transformer-CNN adapter is used to sequentially perform global average pooling (AvgPool2d), 1×1 convolution (Conv1×1), and 1×k convolution on the second intermediate feature map. a Depth stripe convolution (DWConv1×k) a ), k a ×1 depth stripe convolution (DWConv k) a ×1), 1×1 convolution, 3×3 convolution (Conv3×3, stride s=2), BatchNorm (batch normalization), and ReLU activation function processing. Where k a It is a positive integer and can be set according to requirements, for example, set to 7.

[0118] Optional, Figure 9 This is a schematic diagram of the CNN-Transformer adapter according to an embodiment of the present invention. This CNN-Transformer adapter is used for processing fourth intermediate features. Figure 1 Perform global average pooling, 1×1 convolution, and 1×k convolution in the following steps. a Depth stripe convolution, k a ×1 depth stripe convolution, 1×1 convolution, and SiLU activation function processing.

[0119] Optional, Figure 10 This is a schematic diagram of the structure of a convolutional attention module according to an embodiment of the present invention. The convolutional attention module sequentially includes a first batch normalization module (BatchNorm), a self-attention mechanism module, a second batch normalization module (BatchNorm), and a multilayer perceptron module (MLP). Figure 11This is a schematic diagram of the self-attention mechanism module according to an embodiment of the present invention. The self-attention mechanism module includes a first branch and a second branch. By adopting a dual-branch structure, it combines striped convolution and standard convolution, significantly improving attention extraction capabilities while maintaining computational efficiency. When extracting features from the third intermediate feature map using the convolutional attention module, the third intermediate feature map is first standardized using a first-batch normalization module to obtain a first standardized feature map. Then, in the first branch, the first 1×k... d Deep striped convolutional layer, first combined double normalization layer, first k d ×1 depth striped convolutional layer, second combined double normalization layer, first k c ×k c Convolutional layer, third combined double normalization layer and second k c ×k c The convolutional layer processes the first normalized feature map to obtain the first self-attention feature map. In this second branch, the second k-th layer is processed sequentially. d ×1 depth striped convolutional layer, first combined double normalization layer, second 1×k d Deep striped convolutional layer, second combined double normalization layer, third k c ×k c Convolutional layer, third combined double normalization layer and fourth k c ×k c The convolutional layer processes the first normalized feature map to obtain the second self-attention feature map. Where k c The sum is k d A positive integer, which can be set according to actual needs. Then, the first self-attention feature map and the second self-attention feature map are added together to obtain a self-attention feature map. The third intermediate feature map is added together to obtain a summed feature map. This summed feature map is then processed sequentially through the second batch normalization module and the multilayer perceptron module to obtain a self-attention fusion feature map. Finally, the summed feature map is added together with the self-attention fusion feature map to obtain the fourth intermediate feature map. An additional combined double normalization layer (GDN) is inserted between the two convolution types (convolutional layer and depth stripe convolutional layer). Figure 11 It includes a first combined double normalization layer, a second combined double normalization layer, and a third combined double normalization layer, which can reduce the scale sensitivity between input and output.

[0120] Optionally, the encoder consists of three cascaded CBR blocks (which sequentially include a 2D convolutional layer Conv2d, a 2D batch normalization layer BatchNorm2d, and a ReLU activation function) to progressively reduce the resolution and enhance the feature representation.

[0121] Optionally, the decoder consists of three CTBR blocks (which consist of a transposed / deconvolutional layer ConvTranspose2d, a two-dimensional batch normalization layer BatchNorm2d, and a ReLU activation function, respectively), which are responsible for progressively restoring the spatial resolution. Then, the output feature maps Q, S, C, and W are generated through four convolutions.

[0122] Optionally, the upsampling module is a 4x upsampling module.

[0123] For example, the grab detection result is represented as G = {Q, S, C, W}, where Q is the grab quality, S is the sine component of the grab angle, C is the cosine component of the grab angle, and W is the grab width.

[0124] Step S203: Calculate the grasping pose based on the grasping detection result, so that the robot can perform the grasping action on the target to be grasped according to the grasping pose.

[0125] For example, the grasping pose is represented as g = {x, y, q, θ, w}, where (x, y) represents the center coordinates of the grasping rectangle detection box in the image coordinate system, q represents the grasping quality parameter, θ represents the grasping orientation angle, and w represents the width of the grasping rectangle detection box. Further, by locating the position corresponding to the local maximum value in the grasping quality image and combining it with the corresponding grasping angle and width, the optimal grasping position g* is calculated. Then, the 6D pose g corresponding to the optimal grasping position g* in the robot coordinate system is obtained through the extrinsic parameters of the target camera. b As the grasping pose, the robot can perform grasping actions based on the grasping pose.

[0126] The grasping detection method provided in this embodiment, by setting a first Transformer module based on a window multi-head self-attention mechanism, a convolutional attention module, and a second Transformer module based on a multi-head self-attention mechanism connected in sequence in the grasping detection model, can significantly reduce computational complexity while maintaining the ability to model global relationships, improve the ability to perceive local details of the image, preserve the global contextual information of the image, and improve the accuracy and efficiency of grasping detection.

[0127] Furthermore, this embodiment also includes a knowledge distillation-based low-light enhancement module. This module leverages the powerful feature representation capabilities of the low-light enhancement teacher network to improve the performance of the lightweight low-light enhancement student network. The compact low-light enhancement student network performs low-light enhancement processing on RGB images captured in low-light environments, ensuring real-time performance and low parameter count for grasping and detection. This reduces the impact of low-light environments on grasping and detection. Additionally, an illumination gating mechanism is implemented to avoid performance degradation caused by feature space misalignment resulting from directly cascading the low-light enhancement module and the grasping and detection model. This achieves collaborative optimization between the low-light enhancement module and the grasping and detection model, thereby improving the accuracy of grasping and detection.

[0128] As one or more specific application embodiments of the present invention, the optimal implementation scheme or the scheme that the inventors most want to embody is described below in conjunction with specific application scenarios.

[0129] In this embodiment, the input RGB image and depth image are both 224×224 in size; the window size in the first Transformer module is 7×7; in the Transformer-CNN adapter, the input feature map is sequentially processed by global average pooling (AvgPool2d), 1×1 convolution, 1×11 depth stripe convolution, and 11×1 depth stripe convolution; in the CNN-Transformer adapter, the input feature map is sequentially processed by global average pooling, 1×1 convolution, 1×11 depth stripe convolution, 11×1 depth stripe convolution, 1×1 convolution, and SiLU activation function to obtain the module output; in the convolutional attention module, the input feature map in the self-attention mechanism module is processed by two branches respectively: The input feature map of the first branch is sequentially passed through a 1×7 deep stripe convolutional layer, a combined double normalization layer, a 7×1 deep stripe convolutional layer, a combined double normalization layer, a 3×3 standard convolutional layer, a combined double normalization layer, and a 3×3 standard convolutional layer to obtain the output feature map M of the first branch. The input feature map of the second branch is sequentially passed through a 7×1 deep stripe convolutional layer, a combined double normalization layer, a 1×7 deep stripe convolutional layer, a combined double normalization layer, a 3×3 standard convolutional layer, a combined double normalization layer, and a 3×3 standard convolutional layer to obtain the output feature map M2 of the second branch. Feature map M1 and feature map M2 are added together to obtain the output feature map of the self-attention mechanism module. In the knowledge distillation-guided low-light enhancement module, the size of each convolutional layer (Conv) is 3×3.

[0130] Figure 12 This is a schematic diagram of the image involved in the capture and detection process according to an embodiment of the present invention. Figure 12 The data includes RGB images, capture quality feature maps, capture angle feature maps, depth maps, capture detection results, and low-light enhancement results.

[0131] Figure 13 This is a schematic diagram of the grasping pose according to an embodiment of the present invention, wherein (x,y) represents the center coordinates of the grasping rectangle detection box in the image coordinate system, θ represents the grasping direction angle, width represents the width of the grasping rectangle detection box, and height represents the height of the grasping rectangle detection box.

[0132] This embodiment also provides a grasping detection device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0133] This embodiment provides a grasping detection device, such as Figure 14 As shown, it includes:

[0134] The image acquisition module 1401 is used to acquire a target image of the target to be captured; the target image is acquired by the target camera; the target image contains depth information and RGB information.

[0135] The detection module 1402 is used to process the target image through the grasping detection model to obtain the grasping detection result; the grasping detection model includes a first Transformer module, a convolutional attention module and a second Transformer module connected in sequence.

[0136] The pose calculation module 1403 is used to calculate the grasping pose based on the grasping detection result, so that the robot can perform the grasping action on the target to be grasped according to the grasping pose.

[0137] In one optional implementation, the grasping detection model sequentially includes an encoder, a first Transformer module, a Transformer-CNN adapter, a convolutional attention module, a CNN-Transformer adapter, a second Transformer module, a feature fusion module, an upsampling module, and a decoder. The detection module is further configured to: extract features from the target image using the encoder to obtain a first intermediate feature map; process the first intermediate feature map using the first Transformer module based on a window multi-head self-attention mechanism to obtain a second intermediate feature map; and process the second intermediate feature map using the Transformer-CNN adapter. The image is adapted to obtain a third intermediate feature map; features are extracted from the third intermediate feature map using a convolutional attention module to obtain a fourth intermediate feature map; the fourth intermediate feature map is adapted using a CNN-Transformer adapter to obtain a fifth intermediate feature map; the fifth intermediate feature map is processed using a multi-head self-attention mechanism using a second Transformer module to obtain a sixth intermediate feature map; features are fused using a feature fusion module to obtain a fused feature map; the fused feature map is upsampled using an upsampling module to obtain an upsampled feature map; and the upsampled feature map is decoded using a decoder to obtain the capture detection result.

[0138] In one optional implementation, the convolutional attention module includes a first branch and a second branch; the detection module is further configured to: standardize the third intermediate feature map through a first batch of normalization modules to obtain a first standardized feature map; in the first branch, sequentially pass through a first 1×k... d Deep striped convolutional layer, first combined double normalization layer, first k d ×1 depth striped convolutional layer, second combined double normalization layer, first k c ×k c Convolutional layer, third combined double normalization layer and second k c ×k c The convolutional layer processes the first normalized feature map to obtain the first self-attention feature map; in the second branch, it sequentially passes through the second k... d ×1 depth striped convolutional layer, first combined double normalization layer, second 1×k d Deep striped convolutional layer, second combined double normalization layer, third k c ×k c Convolutional layer, third combined double normalization layer and fourth k c ×k c The convolutional layer processes the first normalized feature map to obtain the second self-attention feature map; the first self-attention feature map and the second self-attention feature map are added together to obtain the self-attention feature map; the third intermediate feature map is added together to obtain the added feature map; the added feature map is processed sequentially through the second batch normalization module and the multilayer perceptron module to obtain the self-attention fusion feature map; the added feature map is added together with the self-attention fusion feature map to obtain the fourth intermediate feature map.

[0139] In an optional implementation, the image acquisition module is further configured to: acquire an RGB image and a depth image of the target to be captured using a target camera; convert the RGB image into a grayscale image and calculate the average brightness of the grayscale image; determine whether the average brightness of the grayscale image meets the low-light enhancement conditions; if it does, perform low-light enhancement processing on the RGB image through the low-light enhancement module to obtain an enhanced RGB image; and add the processed RGB image to the depth image to obtain the target image.

[0140] In one optional implementation, the low-light enhancement module includes a low-light enhancement student network and a low-light enhancement teacher network; the image acquisition module is further configured to: perform low-light enhancement processing on the RGB image through the low-light enhancement student network of the low-light enhancement module to obtain an enhanced RGB image; wherein, the low-light enhancement teacher network updates the network weights of the low-light enhancement student network through a loss function corresponding to knowledge distillation.

[0141] In an optional implementation, the image acquisition module is further configured to: extract features from the RGB image through a first convolutional layer to obtain a first feature map; extract features from the first feature map through a second convolutional layer to obtain a second feature map; multiply the RGB image and the second feature map element-wise to obtain a third feature map; process the first feature map through a bias module to obtain a fourth feature map; add the first feature map and the fourth feature map to obtain a fifth feature map; extract features from the fifth feature map through a third convolutional layer to obtain a sixth feature map; add the third feature map and the sixth feature map to obtain a seventh feature map; extract features from the RGB image through a fourth convolutional layer to obtain an eighth feature map; perform correction processing on the eighth feature map through a correction module to obtain a color correction matrix and gamma parameters; process the color correction matrix through a first fully connected layer to obtain a ninth feature map; process the gamma parameters through a second fully connected layer to obtain target parameters; multiply the seventh feature map and the ninth feature map by a matrix to obtain a tenth feature map; and perform an exponential operation on the target parameters and the tenth feature map to obtain an enhanced RGB image.

[0142] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0143] In this embodiment, the grasping and detection device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0144] This invention also provides a computer device having the above-described features. Figure 14 The grabbing and detection device shown.

[0145] Please see Figure 15 , Figure 15 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 15As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 15 Take a processor 10 as an example.

[0146] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0147] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0148] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0149] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0150] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 15 Taking the example of a connection between China and Israel via a bus.

[0151] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0152] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0153] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0154] Although embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and all such modifications and variations fall within the protection scope of the present invention.

Claims

1. A grasping detection method, characterized in that, The capture detection method includes: Acquire a target image of the target to be captured; the target image contains depth information and RGB information; The target image is processed by a grasping detection model to obtain grasping detection results; the grasping detection model includes a first Transformer module, a convolutional attention module and a second Transformer module connected in sequence; The grasping pose is calculated based on the grasping detection results, so that the robot can perform the grasping action on the target to be grasped according to the grasping pose; The grasping detection model sequentially includes an encoder, a first Transformer module, a Transformer-CNN adapter, a convolutional attention module, a CNN-Transformer adapter, a second Transformer module, a feature fusion module, an upsampling module, and a decoder; the process of processing the target image through the grasping detection model to obtain the grasping detection result includes: The target image is subjected to feature extraction by an encoder to obtain a first intermediate feature map; The first intermediate feature map is processed by the first Transformer module using a window-based multi-head self-attention mechanism to obtain the second intermediate feature map; The second intermediate feature map is adapted using a Transformer-CNN adapter to obtain the third intermediate feature map; The third intermediate feature map is used to extract features through a convolutional attention module to obtain a fourth intermediate feature map; The fourth intermediate feature map is adapted using a CNN-Transformer adapter to obtain the fifth intermediate feature map; The fifth intermediate feature map is processed by the second Transformer module based on a multi-head self-attention mechanism to obtain the sixth intermediate feature map; The feature fusion module performs feature fusion on the sixth intermediate feature map to obtain a feature fusion map; The feature fusion map is upsampled using an upsampling module to obtain an upsampled feature map. The upsampled feature map is decoded by a decoder to obtain the capture detection result; The convolutional attention module includes a first branch and a second branch; the step of extracting features from the third intermediate feature map using the convolutional attention module to obtain a fourth intermediate feature map includes: The third intermediate feature map is standardized by the first batch of normalization modules to obtain the first standardized feature map. In the first branch, the first 1×k is passed sequentially. d Deep striped convolutional layer, first combined double normalization layer, first k d ×1 depth striped convolutional layer, second combined double normalization layer, first k c ×k c Convolutional layer, third combined double normalization layer and second k c ×k c The convolutional layer processes the first normalized feature map to obtain the first self-attention feature map; In the second branch, the second k is passed sequentially. d ×1 depth striped convolutional layer, first combined double normalization layer, second 1×k d Deep striped convolutional layer, second combined double normalization layer, third k c ×k c Convolutional layer, third combined double normalization layer and fourth k c ×k c The convolutional layer processes the first normalized feature map to obtain the second self-attention feature map; Add the first self-attention feature map and the second self-attention feature map to obtain the self-attention feature map; The third intermediate feature map and the self-attention feature map are added together to obtain the summed feature map; The additive feature map is processed sequentially through the second batch normalization module and the multilayer perceptron module to obtain the self-attention fusion feature map; The summed feature map is added to the self-attention fusion feature map to obtain the fourth intermediate feature map.

2. The grasping detection method according to claim 1, characterized in that, The process of acquiring the target image of the target to be captured includes: The target camera acquires RGB and depth images of the target to be captured. Convert the RGB image to a grayscale image and calculate the average brightness of the grayscale image; Determine whether the average brightness of the grayscale image meets the low-light enhancement condition; If the conditions are met, the RGB image is processed by the low-light enhancement module to obtain the enhanced RGB image; The processed RGB image is added to the depth image to obtain the target image.

3. The grasping detection method according to claim 2, characterized in that, The low-light enhancement module includes a low-light enhanced student network and a low-light enhanced teacher network; The step of performing low-light enhancement processing on the RGB image using a low-light enhancement module to obtain an enhanced RGB image includes: The RGB image is enhanced by a low-light enhancement student network using a low-light enhancement module to obtain an enhanced RGB image; wherein, the low-light enhancement teacher network updates the network weights of the low-light enhancement student network using a loss function corresponding to knowledge distillation.

4. The grasping detection method according to claim 3, characterized in that, The process of performing low-light enhancement processing on the RGB image through the low-light enhancement student network of the low-light enhancement module to obtain the enhanced RGB image includes: The first feature map is obtained by extracting features from the RGB image through the first convolutional layer; The second feature map is obtained by extracting features from the first feature map through the second convolutional layer; The RGB image is element-wise multiplied with the second feature map to obtain the third feature map; The first feature map is processed by the bias module to obtain the fourth feature map; Add the first feature map to the fourth feature map to obtain the fifth feature map; The sixth feature map is obtained by extracting features from the fifth feature map through the third convolutional layer; The third feature map is added to the sixth feature map to obtain the seventh feature map; The RGB image is used to extract features through the fourth convolutional layer to obtain the eighth feature map; The eighth feature map is corrected using a correction module to obtain a color correction matrix and gamma parameters; The color correction matrix is ​​processed by the first fully connected layer to obtain the ninth feature map; The gamma parameters are processed by a second fully connected layer to obtain the target parameters; Multiply the seventh feature map by the ninth feature map to obtain the tenth feature map; The target parameters are exponentially combined with the tenth feature map to obtain the enhanced RGB image.

5. A grasping and detection device, characterized in that, The device includes: The image acquisition module is used to acquire a target image of the target to be captured; the target image contains depth information and RGB information. The detection module is used to process the target image through a grasping detection model to obtain grasping detection results; the grasping detection model includes a first Transformer module, a convolutional attention module and a second Transformer module connected in sequence; The pose calculation module is used to calculate the grasping pose based on the grasping detection result, so that the robot can perform the grasping action on the target to be grasped according to the grasping pose; The grasping and detection model sequentially includes an encoder, a first Transformer module, a Transformer-CNN adapter, a convolutional attention module, a CNN-Transformer adapter, a second Transformer module, a feature fusion module, an upsampling module, and a decoder; the detection module is further used for: The target image is subjected to feature extraction by an encoder to obtain a first intermediate feature map; The first intermediate feature map is processed by the first Transformer module using a window-based multi-head self-attention mechanism to obtain the second intermediate feature map; The second intermediate feature map is adapted using a Transformer-CNN adapter to obtain the third intermediate feature map; The third intermediate feature map is used to extract features through a convolutional attention module to obtain a fourth intermediate feature map; The fourth intermediate feature map is adapted using a CNN-Transformer adapter to obtain the fifth intermediate feature map; The fifth intermediate feature map is processed by the second Transformer module based on a multi-head self-attention mechanism to obtain the sixth intermediate feature map; The feature fusion module performs feature fusion on the sixth intermediate feature map to obtain a feature fusion map; The feature fusion map is upsampled using an upsampling module to obtain an upsampled feature map. The upsampled feature map is decoded by a decoder to obtain the capture detection result; The convolutional attention module includes a first branch and a second branch; the step of extracting features from the third intermediate feature map using the convolutional attention module to obtain a fourth intermediate feature map includes: The third intermediate feature map is standardized by the first batch of normalization modules to obtain the first standardized feature map. In the first branch, the first 1×k is passed sequentially. d Deep striped convolutional layer, first combined double normalization layer, first k d ×1 depth striped convolutional layer, second combined double normalization layer, first k c ×k c Convolutional layer, third combined double normalization layer and second k c ×k c The convolutional layer processes the first normalized feature map to obtain the first self-attention feature map; In the second branch, the second k is passed sequentially. d ×1 depth striped convolutional layer, first combined double normalization layer, second 1×k d Deep striped convolutional layer, second combined double normalization layer, third k c ×k c Convolutional layer, third combined double normalization layer and fourth k c ×k c The convolutional layer processes the first normalized feature map to obtain the second self-attention feature map; Add the first self-attention feature map and the second self-attention feature map to obtain the self-attention feature map; The third intermediate feature map and the self-attention feature map are added together to obtain the summed feature map; The additive feature map is processed sequentially through the second batch normalization module and the multilayer perceptron module to obtain the self-attention fusion feature map; The summed feature map is added to the self-attention fusion feature map to obtain the fourth intermediate feature map.

6. A computer device, characterized in that, include: A memory and a processor are interconnected, the memory storing computer instructions, and the processor executing the computer instructions to perform the grasping detection method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the grasping and detection method according to any one of claims 1 to 4.

8. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the grasping and detection method according to any one of claims 1 to 4.

Images