Lightweight object detection method and device based on prompt and multi-modal feature fusion

Abstract

The application discloses a kind of light weight capture detection method and equipment based on prompt and multimodal feature fusion, first obtain the color image of target object and the prompt information of target object to user, prompt information is converted into Gaussian heat map, then with color image together into capture detection model, generate the capture posture of target object;Wherein, capture detection model includes double-flow encoder module, residual block and decoder module;Double-flow encoder module includes two encoding branches and several fusion modules arranged between different encoding layers, two encoding branches respectively carry out feature extraction to two input images, and the image features extracted by two encoding branches are injected into color image encoding branch after being fused by fusion module;The fusion features output by double-flow encoder module are finally output after being up-sampled by residual block and decoder module, the capture posture of target object is output finally.The application can effectively improve the flexibility and accuracy of human-computer cooperation under the premise of ensuring real-time.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a lightweight grasping and detection method and device based on cueing and multimodal feature fusion. Background Technology

[0002] In the existing robot grasping and detection algorithm system, generative grasping and detection networks, represented by GR-ConvNet and GG-CNN, occupy an important position. These methods typically employ a fully convolutional network architecture, transforming the regression of grasping pose into a pixel-level prediction task. They have significant advantages in terms of lightweight design and fast inference speed, meeting the real-time computing requirements of embedded devices. However, these traditional methods are essentially blind; the network mechanically outputs a set of all potentially feasible grasping points within the field of view based solely on low-level geometric features and grasping quality scores, lacking a high-level understanding of the semantic information of the objects. This indiscriminate grasping strategy causes robots to be unable to distinguish between targets and background when faced with scenes of multiple stacked objects or coexisting interference, and they are unable to respond to specific user intent commands (such as specifying to grasp a particular object), thus severely limiting their application in scenarios requiring high-level interaction, such as human-robot collaboration.

[0003] To address the aforementioned issues of semantic gaps and intent blind spots, recent research, exemplified by GraspSAM, has introduced prompting mechanisms. These mechanisms allow users to specify the grasping object through clicking, selecting boxes, or natural language descriptions, thereby endowing the robot with the ability to understand user intent. While these large-model-based approaches significantly improve the directionality and intelligence of grasping, they are typically built upon the Vision Transformer (ViT) architecture. The ViT architecture relies on self-attention mechanisms to establish global dependencies. While this enhances the fusion of multimodal features, it also introduces significant computational overhead and model size. This makes it difficult for existing technologies to achieve real-time closed-loop control on computationally limited embedded edge devices, failing to meet the stringent millisecond-level response requirements of robot grasping tasks.

[0004] Therefore, how to maintain the advantages of lightweight networks in terms of high inference speed and low computational consumption, while endowing them with the ability to understand fine-grained user intentions, is a technical problem that urgently needs to be solved in the field of robot grasping. Summary of the Invention

[0005] To address the shortcomings of existing technologies that struggle to simultaneously achieve intent understanding capabilities and lightweight real-time performance, this invention provides a lightweight grasping and detection method and device based on prompts and multimodal feature fusion, which effectively improves the flexibility and accuracy of human-computer collaboration while ensuring real-time performance.

[0006] To achieve the above technical objectives, the present invention adopts the following technical solution: A lightweight grasping detection method based on cues and multimodal feature fusion is proposed. First, a color image of the target object and user cues for the target object are acquired, and the cues are converted into a Gaussian heatmap. Then, the color image and the Gaussian heatmap are fed into a pre-trained grasping detection model to generate the grasping pose of the target object. The grasping detection model includes a dual-stream encoder module, a residual block, and a decoder module. The dual-stream encoder module includes two encoding branches, with several fusion modules set at different encoding layers between the two encoding branches. The two encoding branches extract features from the Gaussian heatmap and the color image, respectively. The fusion module fuses the image features extracted by the two encoding branches and then injects them into the encoding branch of the color image. The fused features output by the dual-stream encoder module are upsampled by the residual block and the decoder module to finally output the grasping posture of the target object.

[0007] Furthermore, the coordinates of several prompt points selected by the user within the target object mask area in the color image are used as prompt information for the user regarding the target object.

[0008] Furthermore, the method for converting the prompt information into a Gaussian heatmap is as follows: The Gaussian kernel function is used to map each cue point to a probability distribution. Then, by taking the maximum value and fusing the probability distributions of all cue points, a Gaussian heatmap is obtained, represented as: ; In the formula, For the first Image coordinates of the prompt point, The number of prompts, Let be the coordinates of any point in the Gaussian heatmap. Image coordinates Pixel values; The standard deviation of the Gaussian kernel is used to control the response range of the cue point.

[0009] Furthermore, each coding branch includes three coding layers. A first fusion module is set between the second coding layers of the two coding branches, and a second fusion module is set between the third coding layers of the two coding branches. The fusion module performs fusion processing on the color image features and Gaussian heatmap features output from the same coding layers of the two coding branches. The fused features obtained by the first fusion module are then added element-wise to the output of the second coding layer of the color image coding branch and output to the third coding layer of the color image coding branch. The fused features obtained by the second fusion module are then added element-wise to the output of the third coding layer of the color image coding branch and output to the residual block.

[0010] Furthermore, the fusion module employs a spatial and channel dual-path attention mechanism, using two-dimensional guided weights generated from Gaussian heatmap features to dynamically enhance the color image features, as shown below: ; In the formula, and These represent the color image features and Gaussian heatmap features input to the fusion module, respectively. This represents the color image feature alignment operation, by It consists of convolutional layers, normalization layers, and ReLU activation functions, used to map color image features to the fusion space and extract basic visual features; This represents the Gaussian heatmap feature encoding operation, which extracts Gaussian heatmap features through a deep structure of 3×3 convolution and dilated convolution, providing semantic and contextual information for weight generation; and These represent spatial attention and channel attention, respectively. Spatial attention generates pixel-level spatial weights through convolution and Sigmoid activation, while channel attention generates channel weights through global average pooling and a bottleneck structure. The pixel-level spatial weights and channel weights are then multiplied element-wise with the aligned color image features to achieve feature reshaping. This indicates the fusion feature output by the fusion module.

[0011] Furthermore, the grasping pose generated by the grasping detection model includes four prediction branches: grasping quality map, grasping width map, grasping angle sine map, and grasping angle cosine map. The loss function for training the grasping detection model is the sum of the losses from these four prediction branches. The loss of each prediction branch is composed of the prediction loss and a spatially weighted sum guided by the Gaussian heatmap, expressed as: ; In the formula: L i For the first The loss of each prediction branch; and They represent the first The nth sample, the th Each prediction branch pairs coordinates The predicted value and the true label; For smoothing L1 loss function; For the first The Gaussian heatmap of each sample is in coordinates Pixel values; These are adjustable hyperparameters used to control the degree of attention bias of the loss function towards the cue information; The batch size for one training session. These represent the height and width of the predicted output image, respectively.

[0012] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor enables the processor to implement the lightweight grasping and detection method based on cueing and multimodal feature fusion as described above.

[0013] To address the technical challenges of existing generative grasping and detection networks lacking user intent understanding capabilities, and the difficulty of implementing real-time closed-loop control on embedded devices due to the large number of parameters and high computational complexity of existing interactive grasping methods, this invention introduces a prompting and Gaussian heatmap encoding mechanism. Prompt information is encoded as a Gaussian heatmap, which serves as prior knowledge and, together with color images, forms a multimodal data input network. This enables the network to accurately respond to user commands in complex environments with multiple objects stacked or otherwise obstructed, generating grasping poses only for specified target objects. This achieves precise guided grasping of user-commanded targets in complex scenarios, enabling efficient human-machine collaborative grasping. Furthermore, by constructing an efficient prompt information encoding mechanism and a guided attention fusion strategy, this invention maintains the advantages of a lightweight architecture and rapid inference. Therefore, this invention effectively improves the robot's target-oriented grasping capability in open scenarios while significantly reducing the number of model parameters and computational costs. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the overall network structure of the present invention.

[0015] Figure 2 This is a schematic diagram of the color image and Gaussian heatmap fusion module of the present invention. Detailed Implementation

[0016] The embodiments of the present invention will be described in detail below. These embodiments are based on the technical solutions of the present invention and provide detailed implementation methods and specific operation processes to further explain the technical solutions of the present invention.

[0017] Example 1

[0018] This invention provides a lightweight grasping detection method based on the fusion of prompts and multimodal features. First, a color image of the target object and the prompt information of the user to the target object are acquired, and the prompt information is converted into a Gaussian heatmap. Then, the color image and the Gaussian heatmap are fed into a pre-trained grasping detection model to generate the grasping posture of the target object.

[0019] The following specific embodiments provide a detailed explanation of the lightweight crawling detection method provided by the present invention, from constructing an image dataset, building a model architecture, training a crawling detection model, and using the crawling detection model to perform actual crawling detection.

[0020] Step 1: Construct the image dataset.

[0021] We obtained the Jacquard image dataset and used 300 object categories from the LVIS dataset (another dataset used for training) as a label pool. We then selected the categories that actually appeared in the Jacquard dataset as valid categories. These valid categories were then sorted from highest to lowest based on their frequency / number of samples in the Jacquard dataset. The data from the top 70% of categories were assigned to the Base set (base class dataset), and the data from the bottom 30% of categories were assigned to the New set (new class dataset). This resulted in a Base set containing 4272 samples and a New set containing 374 samples. The Base set was then divided into training and validation sets in a 9:1 ratio. 90% of the Base set was used for training, and 10% of the Base set and the entire New set were used as the validation set to comprehensively evaluate the training model's performance in grasping both known and unknown objects.

[0022] In addition to color images, the input image data for the prediction model of this invention also requires Gaussian heatmaps obtained based on user prompts. Therefore, it is necessary to convert the user prompts into Gaussian heatmaps. Specifically, in this embodiment, the click prompts are converted into a dense probability field, i.e., a Gaussian heatmap, which is then fused with the color image as an independent feature channel and input into the improved fully convolutional generative network in the subsequent steps for end-to-end grasping pose regression.

[0023] To address the challenge of traditional convolutional neural networks directly processing sparse coordinate points, this invention proposes a method to transform discrete clicks into continuous spatial features. For user-defined clicks on a target object... Coordinates of the prompt point Generate an image with the same size as the input color image. The same single-channel Gaussian heatmap .

[0024] The generation process of a Gaussian heatmap uses a Gaussian kernel function to map cue points to a probability distribution for each pixel in the image. Its pixel value in the Gaussian heatmap Defined as the maximum value of the Gaussian response generated by all cue points at that location, expressed by the following formula: ; in For the first The coordinates of the prompt point, The standard deviation of the Gaussian kernel is used to control the response range of the cue points. Through this step, the user's intent is encoded into a dense feature map with spatial structure information, which serves as prior knowledge to guide the network to focus on the target region.

[0025] To improve the model's robustness to user click position deviations, this invention employs a random sampling strategy during the training phase to construct Gaussian heatmap modal data in the training set. Object mask information is used to randomly sample within the masked region of the target object. Each point serves as a simulated user prompt, and a Gaussian heatmap is dynamically generated for training.

[0026] Step 2: Build the model architecture.

[0027] In order to make full use of the visual information provided by color images and the prior knowledge provided by Gaussian heatmaps, this invention builds a grasping detection model that extracts fused features from color images and Gaussian heatmaps based on a dual-stream encoder.

[0028] This embodiment constructs a grasping detection model comprising a dual-stream encoder module, a residual block, and a decoder module. The dual-stream encoder module includes two encoding branches: the first encoding branch is a color image feature extraction branch, consisting of several encoding layers, each including a convolutional layer, a normalization layer, and an activation function, used to extract target texture, edge, and geometric structure features from the input color image; the second encoding branch is a Gaussian heatmap feature extraction branch, also consisting of several encoding layers, each including a convolutional layer, a normalization layer, and an activation function, used to process the Gaussian heatmap. Several fusion modules are set between the two encoding branches at the same encoding layers. The two encoding branches extract features from the Gaussian heatmap and the color image respectively. The fusion modules fuse the image features extracted by the two encoding branches and then inject them into the color image encoding branch. The fused features output by the dual-stream encoder module are upsampled by the residual block and decoder module to finally output the grasping posture of the target object.

[0029] Specifically, such as Figure 1 As shown, each coding branch includes three coding layers. A first fusion module is set between the second coding layers of the two coding branches, and a second fusion module is set between the third coding layers of the two coding branches. The fusion module performs fusion processing on the color image features and Gaussian heatmap features output from the same coding layers of the two coding branches. The fused features obtained by the first fusion module are then added element-wise to the output of the second coding layer of the color image coding branch and output to the third coding layer of the color image coding branch. The fused features obtained by the second fusion module are then added element-wise to the output of the third coding layer of the color image coding branch and output to the residual block.

[0030] like Figure 2 As shown, each fusion module employs a dual-path attention mechanism (spatial and channel-based), using two-dimensional guided weights generated from Gaussian heatmap features to dynamically enhance the color image features, as represented by: ; In the formula, and These represent the color image features and Gaussian heatmap features input to the fusion module, respectively. This represents the color image feature alignment operation, by It consists of convolutional layers, normalization layers, and ReLU activation functions, used to map color image features to the fusion space and extract basic visual features; This represents the Gaussian heatmap feature encoding operation, which extracts Gaussian heatmap features through a deep structure of 3×3 convolution and dilated convolution, providing semantic and contextual information for weight generation; and These represent spatial attention and channel attention, respectively. Spatial attention generates pixel-level spatial weights through convolution and Sigmoid activation, while channel attention generates channel weights through global average pooling and a bottleneck structure. The pixel-level spatial weights and channel weights are then multiplied element-wise with the aligned color image features to obtain spatially weighted features and channel-weighted features, thus achieving feature reshaping. The fusion feature output by the fusion module is obtained by summing the aligned color image features, spatial weighted features, and channel weighted features.

[0031] The final output of the dual-stream encoder module, the encoded features, are input as residual blocks. After processing through five stacked residual blocks, they are upsampled by transposed convolutions of the three decoding layers of the decoder to predict and output a grasping quality map, a grasping width map, a grasping angle sine map, and a grasping angle cosine map, thus synthesizing the grasping posture. Defined in the image coordinate system as: ; In the formula, The center pixel coordinates of the capture box; The angle by which the mechanical gripper rotates around the z-axis; This refers to the opening width of the grippers; This determines the capture quality score for that pixel.

[0032] Step 3: Train the grab detection model.

[0033] To enhance the crawling detection model's ability to perceive user click locations, this invention introduces a distance-weighted loss mechanism during the training phase. This mechanism dynamically reshapes the spatial loss distribution using the input Gaussian heatmap, enabling the crawling detection model to pay more attention to the prediction error in high-response areas of the prompt point.

[0034] This embodiment will train the total loss function of the grab detection model. Defined as the sum of the weighted losses of each prediction branch: ; Among them, the weighted loss of each prediction branch (grab quality map, grab width map, grab angle sine map, grab angle cosine map) The calculation formula is as follows: ; In the formula: and They represent the first The nth sample, the th Each prediction branch pairs coordinates The predicted value and the true label; For smoothing L1 loss function; For the first The Gaussian heatmap of each sample is in coordinates Pixel values; These are adjustable hyperparameters used to control the degree of attention bias of the loss function towards the cue information; The batch size for one training session. These represent the height and width of the predicted output image, respectively.

[0035] This distance-weighted loss mechanism forces the grasping detection model to assign a higher penalty weight to the area where the target object is located during the optimization process. This not only improves the pose regression accuracy of the grasping detection model on the specified target object, but also effectively solves the background false detection problem that is easily caused by global average loss, ensuring that the robot can accurately execute the target object grasping task given by the user.

[0036] In this embodiment, the experimental environment was configured with an Ubuntu 20.04.6 LTS operating system, and the software environment included Python 3.8.20 and PyTorch 2.0.1+cu117. The hardware platform was equipped with an NVIDIA GeForce RTX 4090 (24G) graphics card, 128G of memory, and a 2T hard drive. During model training, a total of 100 epochs were trained, and the Adam optimizer was used for parameter updates. The initial learning rate was set to 1e-4, and the batch size was set to 8.

[0037] Step 4: Perform actual crawling detection using the crawling detection model.

[0038] First, acquire a color image of the target object and the user's prompts for the target object. Specifically, the coordinates of several prompt points selected by the user within the target object's mask area in the color image—either by clicking or selecting within the mask area—will serve as the user's prompts for the target object.

[0039] Then, the prompt information is converted into a Gaussian heatmap, and the specific conversion method is the same as that used in the construction of the training dataset in step 1.

[0040] The color image of the target object and the converted Gaussian heatmap are then added element-wise along the channel dimension as the multimodal input of the grasping detection model. Feature extraction and fusion are performed through each network layer, and finally the predicted grasping quality map, grasping angle map and grasping width map are output in parallel through the multi-task output head.

[0041] Finally, to comprehensively and quantitatively evaluate the accuracy of the network-predicted grasping pose proposed in this invention, this embodiment establishes strict criteria for determining the success of a single grasp. A predicted grasping box is considered a correct grasp if and only if it simultaneously satisfies the following two geometric constraints:

[0042] 1. The intersection-union ratio (IUU) between the predicted crawl box and the actual label crawl box must be greater than 25%.

[0043] 2. The absolute deviation between the predicted grasping angle and the actual grasping angle must be less than 30°.

[0044] To evaluate the crawling performance of this invention based on prompts and its generalization ability in unknown scenarios, this embodiment employs a category-based testing strategy. The experiment divides the dataset into two independent sets: a Base set and a New set. Testing on the Base set primarily evaluates the model's fitting ability, i.e., verifying whether the model can accurately respond to point prompts and generate high-quality crawling poses under known feature distributions. Testing on the New set aims to evaluate the model's generalization ability, i.e., verifying whether the model can still understand the user's intent and transfer its crawling knowledge when faced with entirely new object shapes and textures.

[0045] Based on the above tests, in order to balance the model's stability in known scenarios with its adaptability in unknown scenarios, this experiment further introduces the harmonic mean as a comprehensive evaluation index, the calculation formula of which is: ; The harmonic mean H can simultaneously maintain the precision of the Base set. ) and New set precision ( By maximizing H, this invention not only pursues high accuracy in familiar environments but also emphasizes robustness in open-world environments, thereby verifying the effectiveness of the Gaussian heatmap guidance strategy in practical robot applications.

[0046] To verify the superiority of the lightweight grasping and detection method of this invention, the GraspSAM network was selected as the comparison model. The experiment was repeated three times, and the experimental results with the highest accuracy on the Base set were taken. A detailed comparison was made from three dimensions: prediction accuracy, number of model parameters, and frame rate. The comparison between GraspSAM and the model used in this method is shown in Table 1.

[0047] Table 1 Comparison between GraspSAM and the method of this invention

[0048] As shown in Table 1, in the Jacquard dataset test, the present invention achieved a grasping accuracy of 87.15% on the Base set, a 3.04% improvement compared to GraspSAM; on the more challenging New set, the grasping accuracy reached 77.54%, a significant 9.63% improvement compared to GraspSAM. The comprehensive evaluation index H reached 82.06%, verifying the excellent generalization performance of the present invention when handling objects of unknown shapes. Therefore, the present invention shows significant improvements in both grasping accuracy and generalization ability. In addition, the grasping and detection model of the present invention has only 2.53M parameters, while the inference frame rate is as high as 209 FPS. While significantly reducing model complexity, it achieves a significant improvement in inference speed, meeting the requirements of millisecond-level real-time closed-loop control, and therefore can be easily deployed in embedded edge devices with limited computing power.

[0049] The above embodiments are preferred embodiments of this application. Those skilled in the art can make various changes or improvements based on them. Without departing from the overall concept of this application, these changes or improvements should fall within the scope of protection claimed in this application.

Claims

1. A lightweight crawling and detection method based on cueing and multimodal feature fusion, characterized in that, First, a color image of the target object and user prompts for the target object are acquired, and the prompts are converted into a Gaussian heatmap. Then, the color image and Gaussian heatmap are fed into a pre-trained grasping detection model to generate the grasping pose of the target object. The grasping detection model includes a dual-stream encoder module, a residual block, and a decoder module. The dual-stream encoder module includes two encoding branches, with several fusion modules set at different encoding layers between the two encoding branches. The two encoding branches extract features from the Gaussian heatmap and the color image, respectively. The fusion module fuses the image features extracted by the two encoding branches and then injects them into the encoding branch of the color image. The fused features output by the dual-stream encoder module are upsampled by the residual block and the decoder module to finally output the grasping posture of the target object.

2. The lightweight grasping and detection method based on cueing and multimodal feature fusion according to claim 1, characterized in that, The coordinates of several prompt points selected by the user within the target object mask area in the color image are used as prompt information for the target object.

3. The lightweight grasping and detection method based on cueing and multimodal feature fusion according to claim 1, characterized in that, The method for converting the prompt information into a Gaussian heatmap is as follows: The Gaussian kernel function is used to map each cue point to a probability distribution. Then, by taking the maximum value and fusing the probability distributions of all cue points, a Gaussian heatmap is obtained, represented as: ； In the formula, For the first Image coordinates of the prompt point, The number of prompts, Let be the coordinates of any point in the Gaussian heatmap. Image coordinates Pixel values; The standard deviation of the Gaussian kernel is used to control the response range of the cue point.

4. The lightweight grasping and detection method based on cueing and multimodal feature fusion according to claim 1, characterized in that, Each coding branch includes three coding layers. A first fusion module is set between the second coding layers of the two coding branches, and a second fusion module is set between the third coding layers of the two coding branches. The fusion module performs fusion processing on the color image features and Gaussian heatmap features output from the same coding layers of the two coding branches. The fused features obtained by the first fusion module are then added element-wise to the output of the second coding layer of the color image coding branch and output to the third coding layer of the color image coding branch. The fused features obtained by the second fusion module are then added element-wise to the output of the third coding layer of the color image coding branch and output to the residual block.

5. The lightweight grasping and detection method based on cueing and multimodal feature fusion according to claim 1, characterized in that, The fusion module employs a spatial and channel dual-path attention mechanism, using dual-dimensional guided weights generated from Gaussian heatmap features to dynamically enhance color image features, as shown below: ； In the formula, and These represent the color image features and Gaussian heatmap features input to the fusion module, respectively. This represents the color image feature alignment operation, by It consists of convolutional layers, normalization layers, and ReLU activation functions, used to map color image features to the fusion space and extract basic visual features; This represents the Gaussian heatmap feature encoding operation, which extracts Gaussian heatmap features through a deep structure of 3×3 convolution and dilated convolution, providing semantic and contextual information for weight generation; and These represent spatial attention and channel attention, respectively. Spatial attention generates pixel-level spatial weights through convolution and Sigmoid activation, while channel attention generates channel weights through global average pooling and a bottleneck structure. The pixel-level spatial weights and channel weights are then multiplied element-wise with the aligned color image features to achieve feature reshaping. This indicates the fusion feature output by the fusion module.

6. The lightweight grasping and detection method based on cueing and multimodal feature fusion according to claim 1, characterized in that, The grasping pose generated by the grasping detection model includes four prediction branches: grasping quality map, grasping width map, grasping angle sine map, and grasping angle cosine map. The loss function for training the grasping detection model is the sum of the losses from these four prediction branches. The loss of each prediction branch is composed of the prediction loss and a spatially weighted sum guided by the Gaussian heatmap, expressed as: ； In the formula: L i For the first The loss of each prediction branch; and They represent the first The nth sample, the th Each prediction branch pairs coordinates The predicted value and the true label; For smoothing L1 loss function; For the first The Gaussian heatmap of each sample is in coordinates Pixel values; These are adjustable hyperparameters used to control the degree of attention bias of the loss function towards the cue information; The batch size for one training session. These represent the height and width of the predicted output image, respectively.

7. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, the processor implements the lightweight crawling and detection method based on cueing and multimodal feature fusion as described in any one of claims 1 to 6.

Images