A deep learning-based intelligent grasping detection method, device and medium for robotic arms

Abstract

This invention relates to a deep learning-based intelligent grasping detection method, device, and medium for a robotic arm. The method builds an ECBAFormer network model, inputs a created image dataset and label file into the network for training, calculates feature classification loss and grasping pose regression loss, and ultimately detects the target category and grasping pose. By simplifying the feature extraction network, a highly efficient and fast feature extraction network architecture, MetaFormer, is derived. This lightweight detection network maintains feature extraction accuracy while maintaining high accuracy. Compared to existing technologies, the present invention offers advantages such as high detection accuracy, wide applicability, and high efficiency.

Description

Technical Field

[0001] The present invention relates to an intelligent control system for a robotic arm, and in particular to a method, device, and medium for intelligent grasping and detection of a robotic arm based on deep learning. Background Art

[0002] Robotic arms are flexible and versatile automation tools with widespread applications and demand in industrial production. Robotic grasping technology is widely used in industrial and service robotics, serving as the foundation for robotic operations such as welding, sorting, packaging, and handling. Currently, industrial robotic grasping operations are primarily controlled through teaching or offline programming, executing planned operations based on manual input. While this approach offers low execution costs, the robot may not function effectively when the environment changes, requiring manual reconfiguration. Consequently, researchers have begun integrating robotics with vision technology. Using visual sensors, robots can obtain environmental information. Using appropriate computer vision algorithms, they can locate target objects, enabling autonomous handling, sorting, loading and unloading, and other operations.

[0003] Currently, mainstream grasping detection algorithms are based on deep learning, or data-driven grasping detection strategies. Data-driven approaches avoid the need for many physical models and reduce computational complexity. Deep learning methods have made significant progress in fields such as machine vision and natural language processing. The development of grasping detection follows, to a certain extent, the path of image detection. Grasping detection models are developed based on improved object detection models. Since robotic grasping is continuous, the grasping poses obtained by grasping detection should also be continuous. The discrete nature of object detection does not fully match the continuity requirement of grasping detection.

[0004] In real-world production environments, robotic arms must perform grasping operations in unstructured environments. Whether stacking or autonomous, ordered grasping, these applications require robust grasp detection and enhanced detection capabilities. In practice, robotic arms are required to detect not only the grasping pose but also the object's category. Therefore, how to simultaneously detect both the grasping pose and category of an object, while achieving more accurate target detection, poses a technical challenge. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a deep learning-based robotic arm intelligent grasping detection method, equipment and medium. By adopting the backbone network of the Metaformer architecture, the background of the target can be filtered out, the impact of the background on grasping detection can be reduced, and the target grasping posture and category can be detected simultaneously, enriching the detection content.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to one aspect of the present invention, a deep learning-based intelligent grasping detection method for a robotic arm is provided, comprising the following steps:

[0008] S1. Use the object to be detected, the grab box label, and the target category label as training input to create an object grab detection and classification dataset and divide it into a training set and a test set.

[0009] S2. Input the training dataset into the ECBAFormer deep learning model network for training. The backbone network extracts features from the input image and generates feature maps and target categories at each stage.

[0010] S3. Input the generated feature map into the SGG-CNN block, and output the center probability, width, sine and cosine values ​​of the grasping angle of each pixel through the convolution and transposed convolution layers;

[0011] S4. Processing the output result, performing non-maximum suppression on the target category information, and determining the category of the predicted target; selecting the five points with the highest probability in the grasping posture information as the center of the grasping box;

[0012] S5. Compare the processed data with the label, calculate the accuracy of the grasp detection, and select an alternative inference model;

[0013] S6. Input the test set image into each candidate inference model, compare the classification accuracy and grasp detection accuracy output by each candidate inference model, and select the model output with the best performance as the detection result.

[0014] The S1 dataset adds category labels of objects to be detected based on the public dataset, expands the number of samples of objects to be detected and grasped of the same category, and labels the expanded samples.

[0015] The ECBAFormer (Efficient Convolutional Block Attention Transformer) deep learning model network for detecting grasp target categories and grasp pose information includes a feature extraction backbone network MetaFormer and a grasp detection pose generation module SGG-CNN, wherein the ECBAM module is embedded in the position of the fusion information unit in the MetaFormer.

[0016] The feature extraction backbone network MetaFormer has the ability to extract features due to the architecture of the network itself, which includes four stages, all of which use downsampling doubling: the first stage includes four MetaFormer Block blocks, the second stage includes four MetaFormer Block blocks, the third stage includes twelve MetaFormer Block blocks, and the fourth stage includes four MetaFormer Block blocks.

[0017] Furthermore, the MetaFormer Block includes two residual sub-blocks. The first residual sub-block includes a fusion information unit for connecting information between tokens; the second residual sub-block includes a two-layer multi-layer perceptron with nonlinear activation for extracting and outputting feature information.

[0018] Furthermore, the grasp detection pose generation module SGG-CNN includes a dimension transformation convolution layer and a transposed convolution layer, which outputs the probability of the center of the grasp box, the width of the grasp box, the sine value of the grasp angle, and the cosine value of the grasp angle.

[0019] The ECBAM module sequentially obtains attention maps in the channel and spatial dimensions and multiplies the attention maps by the input feature maps for adaptive feature training. For the channel attention map, the ECBAM module adaptively captures the dependencies between channels through one-dimensional convolution to obtain the channel attention map. For the spatial attention map, the ECBAM module generates feature descriptors by applying average pooling and maximum pooling operations along the channel axis and connecting the channels. The spatial attention map is generated using a convolutional layer on the descriptor.

[0020] According to a second aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the method when executing the program.

[0021] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] (1) Improve detection efficiency and accuracy: By simplifying the feature extraction network, we obtain an efficient and fast feature extraction network architecture MetaFormer, which is lightweight while ensuring the network's feature extraction capability. The network can also filter out the target background, improving detection efficiency and accuracy.

[0024] (2) Increase the scope of application and improve accuracy: The robot is designed to automatically detect the target object categories, and can accurately detect objects of different categories and shapes, thereby improving the generalization ability of detection.

[0025] (3) Improve the success rate of robotic arm grasping: Through accurate target category and grasping posture detection, the robustness of the robotic arm grasping under different targets and environments is enhanced, and the grasping success rate is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of the deep learning-based intelligent grasping detection method for robotic arms;

[0027] Figure 2 This is the structural diagram of the deep learning model ECBAFormer;

[0028] Figure 3 This is the MetaFormer backbone network model architecture diagram;

[0029] Figure 4 Schematic diagram of the combination of the Channel Attention module and the Spatial Attention module;

[0030] Figure 5 This is the ECBAM module structure diagram. DETAILED DESCRIPTION

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0032] like Figure 1 As shown in FIG, a robot arm intelligent grasping detection method based on deep learning includes the following steps:

[0033] S1. Use the object to be detected, the grab box label, and the target category label as training input to create an object grab detection and classification dataset and divide it into a training set and a test set.

[0034] S2. Input the training dataset into the ECBAFormer deep learning model network for training. The backbone network extracts features from the input image and generates feature maps and target categories at each stage.

[0035] S3. Input the generated feature map into the SGG-CNN block, and output the center probability, width, sine and cosine values ​​of the grasping angle of each pixel through the convolution and transposed convolution layers;

[0036] S4. Processing the output result, performing non-maximum suppression on the target category information, and determining the category of the predicted target; selecting the five points with the highest probability in the grasping posture information as the center of the grasping box;

[0037] S5. Compare the processed data with the label, calculate the accuracy of the grasp detection, and select an alternative inference model;

[0038] S6. Input the test set image into each candidate inference model, compare the classification accuracy and grasp detection accuracy output by each candidate inference model, and select the model output with the best performance as the detection result.

[0039] The dataset S1 of this embodiment adds category labels of objects to be detected based on the public dataset Cornell, expands the number of samples of objects to be detected and grasped of the same category, and labels the expanded samples.

[0040] The input image is first converted into a feature vector through the Patch Embed layer to provide input for subsequent modules. Figure 2 As shown in the figure, the ECBAForm deep learning model network for detecting grasp target categories and grasp pose information includes a feature extraction backbone network MetaFormer and a grasp detection pose generation module SGG-CNN, in which the ECBAM module is embedded in the position of the fusion information unit in MetaFormer.

[0041] like Figure 3 As shown, the feature extraction backbone network MetaFormer does not have a designated fusion information unit; its ability to extract features lies in the network's architecture. The input image is first converted into a feature vector through a block embedding layer, which provides input for subsequent modules. The backbone network consists of four stages, all of which use double downsampling: the first stage includes four MetaFormerBlock blocks, the second stage includes four MetaFormerBlock blocks, the third stage includes twelve MetaFormerBlock blocks, and the fourth stage includes four MetaFormerBlock blocks.

[0042] The MetaFormer Block consists of two incomplete sub-modules. The first incomplete sub-module includes a fusion information unit, which is used to connect the information between tokens; the second incomplete sub-module includes a two-layer multi-layer perceptron with nonlinear activation, which is used to extract and output feature information.

[0043] The grasp detection pose generation module SGG-CNN includes a dimension transformation convolution layer and a transposed convolution layer, which outputs the probability of the center of the grasp box, the width of the grasp box, the sine value of the grasp angle, and the cosine value of the grasp angle.

[0044] like Figure 4 As shown, the ECBAM module sequentially obtains attention maps in the channel and spatial dimensions, and multiplies the attention maps by the input feature maps to perform adaptive feature training, thereby capturing the key information in the features more comprehensively. Through the gradual application of channel attention and spatial attention, ECBAM is able to adaptively adjust the importance of each channel and spatial position in the input feature map. This adaptive recalibration mechanism allows the model to dynamically focus on the most useful features based on task requirements and content context, thereby improving the model's representational capabilities and decision accuracy. For the channel attention map, the ECBAM module adaptively captures the dependencies between channels through one-dimensional convolution to obtain the channel attention map. For the spatial attention map, the ECBAM module generates a feature descriptor by applying average pooling and maximum pooling operations along the channel axis and connecting the channels, and uses a convolutional layer on the descriptor to generate the spatial attention map. As shown Figure 5 Shown is the ECBAM module structure diagram.

[0045] In actual scenarios, the targets of the object categories included in the dataset are placed on the experimental table, and the ECBAFormer model is used as the visual driver to detect the detection accuracy of the model in actual scenarios.

[0046] As shown in Table 1, the method disclosed in this embodiment is compared with other commonly used methods in terms of only detecting the grasping posture and simultaneously detecting the grasping posture and target category. It can be seen that this embodiment has a higher grasping success rate than the existing methods and can also accurately identify the target category.

[0047] Table 1 Comparison of this embodiment and other experimental models on grasping posture and target category detection

[0048]

[0049]

[0050] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0051] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0052] Multiple components in the device are connected to the I / O interface, including: input units, such as a keyboard, mouse, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as magnetic disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunications networks. The processing unit performs the various methods and processes described above, such as the method of the present invention. For example, in some embodiments, the method of the present invention can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device via ROM and / or the communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the method of the present invention described above can be performed. Alternatively, in other embodiments, the CPU can be configured to perform the method of the present invention by any other suitable means (e.g., by means of firmware).

[0053] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0054] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0055] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0056] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A deep learning-based intelligent grasping detection method for a robotic arm, characterized in that: The steps include: S1. Use the object to be detected, the grab box label, and the target category label as training input to create an object grab detection and classification dataset and divide it into a training set and a test set. S2. Input the training dataset into the ECBAFormer deep learning model network for training. The backbone network extracts features from the input image and generates feature maps and target categories at each stage. S3. Input the feature map into the SGG-CNN block, and output the center probability, width, sine and cosine values ​​of the grabbing angle at each pixel through the convolution and transposed convolution layers; S4. Processing the output result, performing non-maximum suppression on the target category information, and determining the category of the predicted target; Select the five points with the highest probability in the grasping pose information as the center of the grasping box; S5. Compare the processed output results with the labels, calculate the accuracy of the grasp detection, and select an alternative inference model; S6. Input the test set images into each candidate inference model, compare the classification accuracy and grasp detection accuracy output by each candidate inference model, and select the model output with the best performance as the detection result; The ECBAFormer deep learning model network for detecting grasping target categories and grasping posture information includes a feature extraction backbone network MetaFormer and a grasping detection posture generation module SGG-CNN, wherein the ECBAM module is embedded in the position of the fusion information unit in MetaFormer.

2. The deep learning-based intelligent grasping detection method for a robotic arm according to claim 1, characterized in that: The feature extraction backbone network MetaFormer consists of four stages, all of which use downsampling doubling; The first stage includes four MetaFormer Block blocks, the second stage includes four MetaFormer Block blocks, the third stage includes twelve MetaFormer Block blocks, and the fourth stage includes four MetaFormer Block blocks.

3. The deep learning-based intelligent grasping detection method for a robotic arm according to claim 2, characterized in that: The MetaFormer Block includes two incomplete sub-modules. The first incomplete sub-module includes a fusion information unit for connecting information between tokens; the second incomplete sub-module includes a two-layer multi-layer perceptron with nonlinear activation for extracting and outputting feature information.

4. The deep learning-based intelligent grasping detection method for a robotic arm according to claim 1, characterized in that: The grasp detection pose generation module SGG-CNN includes a dimension transformation convolution layer and a transposed convolution layer, and outputs the probability of the center of the grasp box, the width of the grasp box, the sine value of the grasp angle, and the cosine value of the grasp angle.

5. The deep learning-based intelligent grasping detection method for a robotic arm according to claim 1, characterized in that: The ECBAM module sequentially obtains attention maps in channel and spatial dimensions, and multiplies the attention maps by the input feature maps for adaptive feature training.

6. The deep learning-based intelligent grasping detection method for a robotic arm according to claim 5, characterized in that: The ECBAM module adaptively captures the dependencies between channels through one-dimensional convolution, thereby obtaining an attention map in the channel dimension.

7. The deep learning-based intelligent grasping detection method for a robotic arm according to claim 5, characterized in that: The ECBAM module generates feature descriptors by applying average pooling and maximum pooling operations along the channel axis and concatenating the channels, and then uses convolutional layers on the descriptors to generate attention maps in the spatial dimension.

8. The deep learning-based intelligent grasping detection method for a robotic arm according to claim 1, characterized in that: The S1 dataset adds category labels of objects to be detected based on the public dataset, expands the number of samples of objects to be detected and grasped of the same category, and labels the expanded samples.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images