Robot grabbing pose detection method and system based on interpretable neural network

Abstract

The invention discloses a robot grabbing pose detection method and system based on an interpretable neural network, and the method comprises the steps: obtaining a three-channel RGB image of a target object, carrying out the image data preprocessing, and obtaining the preprocessed three-channel RGB image of the target object; an NAS technology and jump connection are introduced, a residual error convolution module is combined, and a target object pose detection model is constructed; and on the basis of a target object pose detection model, performing grabbing pose detection on the preprocessed three-channel RGB image of the target object to obtain a robot grabbing pose detection result. According to the invention, the structure or parameters of the neural network can be automatically adjusted according to the external physical characteristics of diversified objects, so that the accuracy of target object pose detection is improved. The robot grabbing pose detection method and system based on the interpretable neural network can be widely applied to the technical field of robot pose detection.

Description

Technical Field

[0001] This invention relates to the field of robot pose detection technology, and in particular to a robot grasping pose detection method and system based on interpretable neural networks. Background Technology

[0002] In robotic grasping tasks, deep learning network models are primarily used to predict the grasping pose of target objects. Manually designed robot pose detection networks can only handle a few or a certain type of objects. However, in industrial settings, with a large number of objects on conveyor belts, their contours, textures, and glosses vary greatly. Influenced by these changes in external physical characteristics, the data distribution of the training model becomes inconsistent with the data in actual application—a phenomenon generally known as "concept drift." The model's performance degrades as it struggles to adapt to the external features of each object. When the external features of an object deviate from the model's training range, it significantly impacts the grasping success rate and challenges the accuracy and stability of the system's recognition. Concept drift necessitates timely adjustments to the neural network's architecture and parameters to adapt to data changes—a complex and time-consuming task. This problem is particularly challenging for production personnel lacking specialized knowledge. Therefore, in the process of robotic grasping, finding effective strategies to adjust the neural network's architecture and parameters and apply them to stable robot grasping has become a critical challenge, addressing the phenomenon of concept drift. Deep neural networks, with their superior feature extraction and pattern recognition capabilities, have achieved breakthroughs in many fields, including image recognition, natural language processing, and decision support systems. However, these models typically contain millions of parameters and complex nonlinear structures, making their decision-making process like a "black box," difficult for humans to understand and interpret. This opacity not only affects people's trust in the model but may also raise questions about the rationality and fairness of its decisions. Summary of the Invention

[0003] To address the aforementioned technical problems, the present invention aims to provide a robot grasping pose detection method and system based on interpretable neural networks, which can automatically adjust the structure or parameters of the neural network according to the external physical characteristics of diverse objects, thereby improving the accuracy of target object pose detection.

[0004] The first technical solution adopted in this invention is: a robot grasping pose detection method based on interpretable neural networks, comprising the following steps: The three-channel RGB image of the target object is acquired and the image data is preprocessed to obtain the preprocessed three-channel RGB image of the target object. By introducing NAS technology and skip connections, and combining it with residual convolution modules, a target object pose detection model is constructed. Based on the target object pose detection model, the robot's grasping pose is detected by performing grasping pose detection on the preprocessed three-channel RGB image of the target object, and the robot grasping pose detection result is obtained.

[0005] Furthermore, the target object pose detection model specifically includes a downsampling feature extraction module, a residual convolution module, and an upsampling feature extraction module. The downsampling feature extraction module, the residual convolution module, and the upsampling feature extraction module are connected in sequence. The downsampling feature extraction module and the upsampling feature extraction module are connected by skip connections. The downsampling feature extraction module and the upsampling feature extraction module have the same structure. The residual convolution module has three residual convolution blocks.

[0006] Furthermore, the downsampling feature extraction module includes a first normal cell search space, an attention search space, a second normal cell search space, and a decaying cell search space. The first normal cell search space includes a first normal cell and a second normal cell. The attention search space includes a first attention cell and a second attention cell. The second normal cell search space includes a third normal cell and a fourth normal cell. The decaying cell search space includes a first decaying cell and a second decaying cell. Both the first attention cell and the second attention cell include an efficient multi-scale attention layer, a squeeze-excitation attention layer, and a convolutional block attention layer. The first normal cell, the second normal cell, the first attention cell, the first decaying cell, the third normal cell, the fourth normal cell, the second attention cell, and the second decaying cell are connected sequentially.

[0007] Furthermore, the residual convolution module includes a first convolutional layer, a segmentation layer, a first convolutional block, a first residual block, a second convolutional block, a second residual block, a merging layer, and a second convolutional layer. The output of the first convolutional layer is connected to the input of the segmentation layer. The output of the segmentation layer is connected to the input of the first convolutional block and the input of the second convolutional block, respectively. The input of the first convolutional block is connected to the input of the first residual block. The output of the second convolutional block is connected to the input of the second residual block. The outputs of the first residual block and the second residual block are both connected to the input of the merging layer. The output of the merging layer is connected to the input of the second convolutional layer.

[0008] Furthermore, the adaptive search space reduction strategy of the downsampling feature extraction module and the upsampling feature extraction module is as follows: Based on the initial space, a random search is performed to obtain the accuracy of each cell and the structure is sorted to obtain the effectiveness probability of each cell. Cells that do not meet the preset requirements are removed based on their effectiveness probability, resulting in an optimized search space. The contribution of each cell in the optimized search space is obtained by using a class activation heatmap. If the contribution does not meet the preset requirements, the adaptive search space reduction strategy is re-executed until the contribution meets the preset requirements, and the final search space is output.

[0009] Furthermore, the step of performing grasping pose detection on the preprocessed three-channel RGB image of the target object based on the target object pose detection model to obtain the robot grasping pose detection result specifically includes: The preprocessed three-channel RGB image of the target object is input into the target object pose detection model; The downsampling feature extraction module based on the target object pose detection model performs global and local feature extraction on the preprocessed three-channel RGB image of the target object to obtain the target object image feature information. The residual convolution module based on the target object pose detection model performs fine-grained feature processing on the target object image feature information to obtain fine-grained feature information of the target object image. The upsampling feature extraction module based on the target object pose detection model performs feature enhancement processing on the fine-grained feature information of the target object image to obtain angle image, quality image and width image; By combining angle images, quality images, and width images, the robot's grasping pose detection results can be obtained.

[0010] The second technical solution adopted in this invention is: a robot grasping pose detection system based on an interpretable neural network, comprising: The first module is used to acquire the three-channel RGB image of the target object and perform image data preprocessing to obtain the preprocessed three-channel RGB image of the target object. The second module is used to introduce NAS technology and skip connections, and combine them with the residual convolution module to build a target object pose detection model. The third module is used to perform grasping pose detection on the preprocessed three-channel RGB image of the target object based on the target object pose detection model, and obtain the robot grasping pose detection result.

[0011] The beneficial effects of the method and system of this invention are as follows: This invention acquires a three-channel RGB image of a target object and performs image data preprocessing to obtain a preprocessed three-channel RGB image of the target object. It further introduces NAS technology and skip connections, combined with a residual convolution module, to construct a target object pose detection model. Using NAS technology, a network architecture is constructed by stacking normal cells, decaying cells, and attention cells to extract image feature information. A residual convolution module is used to further extract fine-grained features, and a skip connection is added to prevent gradient vanishing and enhance the model's expressive power. Finally, based on the target object pose detection model, the preprocessed three-channel RGB image of the target object is used for grasping pose detection to obtain the robot grasping pose detection result, thereby improving the accuracy of target object pose detection. Attached Figure Description

[0012] Figure 1 This is a flowchart of the steps of the robot grasping pose detection method based on an interpretable neural network according to the present invention; Figure 2 This is a structural block diagram of the robot grasping pose detection system based on an interpretable neural network according to the present invention. Figure 3 This is a schematic diagram of the target object pose detection model provided in a specific embodiment of the present invention; Figure 4 This is a schematic diagram of the NAS network structure provided in a specific embodiment of the present invention; Figure 5 This is a schematic diagram of the interpretable adaptive search space reduction strategy provided in a specific embodiment of the present invention; Figure 6 This is a schematic diagram of CAM visualization of each cell in the model provided in a specific embodiment of the present invention; Figure 7 This is a schematic diagram of the residual convolution module provided in a specific embodiment of the present invention. Detailed Implementation

[0013] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0014] First, it's important to clarify that neural architecture search (NAS) is an automated technique used to generate efficient convolutional neural network (CNN) architectures for detection tasks. The core process of NAS consists of three main parts: constructing the search space, selecting a search strategy, and developing a performance evaluation scheme. The search space encompasses all possible network structures and is typically constructed in two ways: one is to define the search space based on existing neural network structures, thus searching the entire network; the other is to construct the search space based on cells to find the optimal cell structure. The search strategy involves finding the best network structure within this space, with common methods including reinforcement learning or gradient descent. The performance evaluation strategy is used to predict and measure the generalization ability of the sampled network structures during the NAS process. Its core goal is to reveal the internal workings of the model, providing an intuitive explanation of the model's decision-making process. This type of research can elucidate how the model extracts features from input data and makes decisions, effectively mitigating the black-box problem. Notably, based on the stronger semantic nature of deep activations in the hierarchical structure of convolutional neural networks, researchers have proposed an intuitive attribution method for image data—Class Activation Map (CAM). CAM is essentially a heatmap that visually demonstrates the basis for neural network decisions by highlighting key regions in the input image that are relevant to a specific category, providing a powerful tool for model interpretation.

[0015] However, existing technologies have the following problems: 1) High time cost: Traditional pose detection methods rely on manual design, which requires a lot of time to build suitable neural networks.

[0016] 2) Insufficient generalization: Traditional algorithms can usually only handle a few specific types of targets. When faced with diverse objects, robots may have difficulty accurately identifying these targets and generating corresponding pose information, which may lead to failures in grasping.

[0017] 3) Poor interpretability: The decision-making process of deep network models is like a "black box". It is impossible to know the degree of influence of the structural blocks of the search network on a specific scenario. The training results are difficult for humans to understand and believe.

[0018] Reference Figure 1 This invention provides a robot grasping pose detection method based on an interpretable neural network, which includes the following steps: S100: Acquire the three-channel RGB image of the target object and perform image data preprocessing to obtain the preprocessed three-channel RGB image of the target object; In this embodiment, the present invention employs an Intel RealSense D435i camera, utilizing its active infrared binocular stereo ranging function to capture RGB images of the target object. The acquired images are used to create a training dataset, and data augmentation techniques, such as image flipping, cropping, and translation, are used to expand the size of the dataset.

[0019] S200 introduces NAS technology and skip connections, and combines residual convolution modules to construct a target object pose detection model; In this embodiment, the target object pose detection model specifically includes a downsampling feature extraction module, a residual convolution module, and an upsampling feature extraction module. The downsampling feature extraction module, the residual convolution module, and the upsampling feature extraction module are connected in sequence. The downsampling feature extraction module and the upsampling feature extraction module are connected by skip connections. The downsampling feature extraction module and the upsampling feature extraction module have the same structure. The residual convolution module has three residual convolution blocks.

[0020] Among them, such as Figure 4 As shown, the downsampling feature extraction module includes a first normal cell search space, an attention search space, a second normal cell search space, and a decaying cell search space. The first normal cell search space includes a first normal cell and a second normal cell. The attention search space includes a first attention cell and a second attention cell. The second normal cell search space includes a third normal cell and a fourth normal cell. The decaying cell search space includes a first decaying cell and a second decaying cell. Both the first attention cell and the second attention cell include an efficient multi-scale attention layer, a squeeze-excitation attention layer, and a convolutional block attention layer. The first normal cell, the second normal cell, the first attention cell, the first decaying cell, the third normal cell, the fourth normal cell, the second attention cell, and the second decaying cell are connected sequentially.

[0021] like Figure 7 As shown, the residual convolution module includes a first convolutional layer, a segmentation layer, a first convolutional block, a first residual block, a second convolutional block, a second residual block, a merging layer, and a second convolutional layer. The output of the first convolutional layer is connected to the input of the segmentation layer. The output of the segmentation layer is connected to the input of the first convolutional block and the input of the second convolutional block, respectively. The input of the first convolutional block is connected to the input of the first residual block. The output of the second convolutional block is connected to the input of the second residual block. The outputs of the first residual block and the second residual block are both connected to the input of the merging layer. The output of the merging layer is connected to the input of the second convolutional layer.

[0022] S300: Based on the target object pose detection model, the robot performs grasping pose detection on the preprocessed three-channel RGB image of the target object to obtain the robot grasping pose detection result.

[0023] S310. Input the preprocessed three-channel RGB image of the target object into the target object pose detection model; S320, a downsampling feature extraction module based on the target object pose detection model, performs global and local feature extraction on the preprocessed three-channel RGB image of the target object to obtain the target object image feature information; In this embodiment, the diversity of objects and complex background environments in the real world pose challenges to object recognition. Traditional convolutional operations often ignore the multi-scale features of objects, making it difficult to effectively filter out background interference. To accurately identify diverse objects in complex environments, we integrate NAS technology into the downsampling feature extraction module. Through automated neural network architecture design, we accurately identify specific targets and generate pose information, which enhances the model's feature extraction capabilities. The downsampling feature extraction module extracts image features while gradually reducing the image size. Therefore, we sample... Figure 4 The module is constructed using the cell stacking method shown. Unlike other methods, we introduce an attention search space, inserting an attention cell between normal cells and decaying cells to address the feature information redundancy caused by stacking multiple normal cells.

[0024] NAS network architecture such as Figure 4 As shown, it includes normal cells, reduce cells, attention cells, and input / output. Normal cells maintain the same feature map size, while reduce cells halve the size. The processing flow of the NAS network is as follows: the input image first passes through two normal cells, then one attention cell, then one reduce cell, then two more normal cells, one attention cell, and one more reduce cell, finally outputting the result. The first two normal cells are searched from the first normal cell search space, the last two from the second normal cell search space, the reduce cell from the reduce cell search space, and the attention cell from the attention search space. The attention search space includes EMA (Efficient Multi-Scale Attention), SE (Squeeze-Activate Attention), and CBAM (Convolutional Block Attention). The cell structures in our defined search space include basic operations such as 1×1 convolution, 3×3 convolution, 5×5 convolution, 3×3 depthwise convolution, normalization, activation functions, and max pooling.

[0025] Typically, manually defined initial search spaces are broad and demanding on computational resources. To improve search efficiency, we employ an interpretable adaptive search space reduction strategy. This strategy automatically evaluates cell performance and the influence of visualized structures in the initial space, gradually eliminating underperforming cells to ultimately form a smaller and more efficient search space, thereby improving the search efficiency and performance of NAS.

[0026] This strategy is as follows Figure 5 As shown, the specific process includes: First, a random search is performed within an initial space. The accuracy of each cell is calculated, and the structures are sorted according to their accuracy. Their effectiveness probability (the probability of appearing in all high-performing networks) is calculated, and poorly performing cells are removed, resulting in an optimized search space. However, a considerable number of cells still exist within this search space. To further filter for efficient cells, we use a class activation heatmap (CAM) to reflect the contribution of each cell to the overall model. If the task requirements are not met, we restart from the first step; if they are met, we obtain the efficient search space we ultimately need.

[0027] Class Activation Maps (CAMs) are techniques used to visualize the decision-making process of deep learning models, primarily in Convolutional Neural Networks (CNNs), to help interpret the regions the model focuses on in image classification tasks. For example... Figure 6 As shown, AJ is a visualization of the contribution of each cell to the overall model. These visualizations reveal which cells contribute more to the model's focus on specific features. This strongly supports the "black box" phenomenon of neural networks and provides a degree of interpretability.

[0028] S330, a residual convolution module based on the target object pose detection model, performs fine-grained feature processing on the target object image feature information to obtain fine-grained feature information of the target object image; In this embodiment, after the downsampling module, the image size becomes 56×56. Further extraction and fusion of effective feature information via consecutive RC modules can then be achieved without changing the image size. The structure diagram of the RC module is shown below. Figure 7 As shown, the data first undergoes a 1×1 convolution, then is divided into two parallel channels, each passing through a convolutional block and a residual block. Finally, the features from the two channels are merged, and then passed through another 1×1 convolution. The 1×1 convolution is used to transform the channel size, while the purpose of splitting into two channels is to extract feature information from different channels and strengthen the correlation between them.

[0029] S340, Upsampling feature extraction module based on target object pose detection model, performs feature enhancement processing on fine-grained feature information of target object image to obtain angle image, quality image and width image; The S350 combines angle images, quality images, and width images to achieve robot grasping pose detection results.

[0030] In this embodiment, after the input image passes through the downsampling feature extraction module and the RC module, its resolution is reduced from 224×224 to 56×56. This size reduction helps the network focus on extracting higher-level features, but it may also lead to the loss of image details. To improve the image's resolvability and preserve its spatial features, we also use NAS technology to automatically design the network architecture for image upsampling, just like the downsampling feature extraction module.

[0031] In summary, as Figure 3 As shown, this embodiment of the invention achieves the task of detecting and generating the pose of a target object through a downsampling feature extraction module (DownsamplingBlock), a residual-conv block (RC) module, and an upsampling feature extraction module (UpsamplingBlock). First, a 224×224 pixel three-channel RGB image is used as input. Then, the feature extraction modules obtain the global and local features of the image. In the UpsamplingBlock and DownsamplingBlock stages, we first utilize NAS technology to construct a network architecture by stacking normal cells, decaying cells, and attention cells to extract the image's feature information. Between the UpsamplingBlock and DownsamplingBlock, we further extract fine-grained features through three RC modules, and we add a skip connection to prevent gradient vanishing and enhance the model's expressive power. After passing through the above modules, three images are finally obtained: an angle image, a quality image, and a width image, thus ultimately detecting the object's grasping pose.

[0032] Therefore, the embodiments of the present invention have the following advantages compared with the prior art: 1) Saves a lot of time: This patent uses neural network architecture search technology to realize the automated construction of robot pose detection network. This not only reduces the workload of manual design, but also improves the performance and efficiency of the network, thereby saving a lot of time and cost.

[0033] 2) High model generalization: Compared to traditional methods, this patent can automatically adjust the structure or parameters of the neural network based on the diverse external physical characteristics of objects to improve the accuracy of target object pose detection. Due to its adaptability, it can be applied to more objects and scenes, exhibiting a certain degree of generalization.

[0034] 3) Strong interpretability: Compared with traditional methods, this patent adopts a similar activation heatmap format. The visual graph directly reflects the cell's sensitivity to specific regions, solving the "black box" problem of neural networks and having a certain degree of interpretability.

[0035] Reference Figure 2 A robot grasping pose detection system based on interpretable neural networks includes: The first module 201 is used to acquire a three-channel RGB image of the target object and perform image data preprocessing to obtain a preprocessed three-channel RGB image of the target object. The second module 202 is used to introduce NAS technology and skip connections, and combine them with the residual convolution module to build a target object pose detection model; The third module 203 is used to perform grasping pose detection on the preprocessed three-channel RGB image of the target object based on the target object pose detection model, and obtain the robot grasping pose detection result.

[0036] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0037] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A robot grasping pose detection method based on interpretable neural networks, characterized in that, Includes the following steps: The three-channel RGB image of the target object is acquired and the image data is preprocessed to obtain the preprocessed three-channel RGB image of the target object. By introducing NAS technology and skip connections, and combining it with residual convolution modules, a target object pose detection model is constructed. Based on the target object pose detection model, the robot's grasping pose is detected by performing grasping pose detection on the preprocessed three-channel RGB image of the target object, and the robot grasping pose detection result is obtained.

2. The robot grasping pose detection method based on interpretable neural networks according to claim 1, characterized in that, The target object pose detection model specifically includes a downsampling feature extraction module, a residual convolution module, and an upsampling feature extraction module. The downsampling feature extraction module, the residual convolution module, and the upsampling feature extraction module are connected in sequence. The downsampling feature extraction module and the upsampling feature extraction module are connected by skip connections. The downsampling feature extraction module and the upsampling feature extraction module have the same structure. The residual convolution module has three residual convolution blocks.

3. The robot grasping pose detection method based on interpretable neural networks according to claim 2, characterized in that, The downsampling feature extraction module includes a first normal cell search space, an attention search space, a second normal cell search space, and a decaying cell search space. The first normal cell search space includes a first normal cell and a second normal cell. The attention search space includes a first attention cell and a second attention cell. The second normal cell search space includes a third normal cell and a fourth normal cell. The decaying cell search space includes a first decaying cell and a second decaying cell. Both the first attention cell and the second attention cell include an efficient multi-scale attention layer, a squeeze-excitation attention layer, and a convolutional block attention layer. The first normal cell, the second normal cell, the first attention cell, the first decaying cell, the third normal cell, the fourth normal cell, the second attention cell, and the second decaying cell are connected sequentially.

4. The robot grasping pose detection method based on interpretable neural networks according to claim 3, characterized in that, The residual convolution module includes a first convolutional layer, a segmentation layer, a first convolutional block, a first residual block, a second convolutional block, a second residual block, a merging layer, and a second convolutional layer. The output of the first convolutional layer is connected to the input of the segmentation layer. The output of the segmentation layer is connected to the input of the first convolutional block and the input of the second convolutional block, respectively. The input of the first convolutional block is connected to the input of the first residual block. The output of the second convolutional block is connected to the input of the second residual block. The outputs of the first residual block and the second residual block are both connected to the input of the merging layer. The output of the merging layer is connected to the input of the second convolutional layer.

5. The robot grasping pose detection method based on interpretable neural networks according to claim 4, characterized in that, The adaptive search space reduction strategy of the downsampling feature extraction module and the upsampling feature extraction module is as follows: Based on the initial space, a random search is performed to obtain the accuracy of each cell and the structure is sorted to obtain the effectiveness probability of each cell. Cells that do not meet the preset requirements are removed based on their effectiveness probability, resulting in an optimized search space. The contribution of each cell in the optimized search space is obtained by using a class activation heatmap. If the contribution does not meet the preset requirements, the adaptive search space reduction strategy is re-executed until the contribution meets the preset requirements, and the final search space is output.

6. The robot grasping pose detection method based on interpretable neural networks according to claim 5, characterized in that, The step of performing grasping pose detection on the preprocessed three-channel RGB image of the target object based on the target object pose detection model to obtain the robot grasping pose detection result specifically includes: The preprocessed three-channel RGB image of the target object is input into the target object pose detection model; The downsampling feature extraction module based on the target object pose detection model performs global and local feature extraction on the preprocessed three-channel RGB image of the target object to obtain the target object image feature information. The residual convolution module based on the target object pose detection model performs fine-grained feature processing on the target object image feature information to obtain fine-grained feature information of the target object image. The upsampling feature extraction module based on the target object pose detection model performs feature enhancement processing on the fine-grained feature information of the target object image to obtain angle image, quality image and width image; By combining angle images, quality images, and width images, the robot's grasping pose detection results can be obtained.

7. A robot grasping pose detection system based on interpretable neural networks, characterized in that, Includes the following modules: The first module is used to acquire the three-channel RGB image of the target object and perform image data preprocessing to obtain the preprocessed three-channel RGB image of the target object. The second module is used to introduce NAS technology and skip connections, and combine them with the residual convolution module to build a target object pose detection model. The third module is used to perform grasping pose detection on the preprocessed three-channel RGB image of the target object based on the target object pose detection model, and obtain the robot grasping pose detection result.

Images