Mechanical arm intelligent grabbing detection method and device based on deep learning and medium

Abstract

The invention relates to a mechanical arm intelligent grabbing detection method and device based on deep learning and a medium. The method comprises the steps that an ECBAFormer network model is built, an established image data set and a label file are input into a network for training, classification loss calculation and grabbing pose regression loss calculation are conducted on features, and finally the category and grabbing pose of a target are detected. The efficient and rapid feature extraction network architecture MetaFormer is obtained by simplifying the feature extraction network, and the accuracy of feature extraction is ensured while the weight of the detection network is reduced. Compared with the prior art, the method has the advantages of high detection accuracy, wide application range, high efficiency and the like.

Description

Technical Field

[0001] The present invention relates to a robot arm intelligent control system, and in particular to a robot arm intelligent grasping detection method, equipment and medium based on deep learning. Background Art

[0002] The robotic arm is a flexible and versatile automation tool that has a wide range of applications and demands in industrial production. Robotic grasping technology is widely used in the fields of industrial robots and service robots, and is the basis for robots to perform operations such as welding, sorting, packaging, and handling. At present, the grasping operation of robots in industry is mainly controlled by teaching or offline programming, and the planned operations are performed according to manual input. Although the execution cost of this method is low, when the environment changes, the robot will not work well and needs to be manually reset. Therefore, researchers began to combine robotics technology with visual technology. Through visual sensors, robots can obtain environmental information, and then use corresponding computer vision algorithms to locate the target object, so that the robot can autonomously carry out operations such as handling, sorting, loading and unloading.

[0003] At present, the mainstream grasping detection algorithms are all based on deep learning, that is, data-driven grasping detection strategies. The data-driven method avoids the establishment of many physical models and reduces the complexity of calculations. Deep learning methods have made significant progress in the fields of machine vision and natural language processing. The development of grasping detection is to a certain extent the development process of image detection. The grasping detection model is improved on the basis of the target detection model. Since the grasping of the robot arm is continuous, the grasping posture obtained by grasping detection should also be continuous. The discreteness of target detection does not fully match the requirement of the continuity of grasping detection.

[0004] In actual production environments, robotic arms need to perform grasping operations in unstructured environments. Whether it is a stacking scenario or autonomous and orderly grasping, the grasping detection of the robotic arm is required to have better robustness and stronger detection capabilities. In practice, the robotic arm is required to detect not only the grasping posture of the object, but also the category of the object. Therefore, how to enable the robotic arm to detect the grasping posture and category of the target at the same time and detect the target more accurately has become a technical problem that needs to be solved. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a deep learning-based robotic arm intelligent grasping detection method, equipment and medium. By adopting the backbone network of the Metaformer architecture, the background of the target can be filtered out, the impact of the background on the grasping detection can be reduced, and the target grasping posture and category can be detected at the same time, enriching the detection content.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to one aspect of the present invention, a robot arm intelligent grasping detection method based on deep learning is provided, comprising the following steps:

[0008] S1. Take the object to be detected, the grab box label and the target category label as training input, create the object grab detection and classification dataset, and divide it into training set and test set;

[0009] S2. Input the training data set into the ECBAFormer deep learning model network for training. The backbone network extracts features from the input image to generate feature maps and target categories at each stage.

[0010] S3, input the generated feature map into the SGG-CNN block, and output the center probability, width, sine and cosine values ​​of the grabbing angle of each pixel through the convolution and transposed convolution layers;

[0011] S4, processing the output result, performing non-maximum suppression on the target category information, and determining the category of the predicted target; selecting five points with the highest probability in the grasping posture information as the center of the grasping box;

[0012] S5, comparing the processed data with the label, calculating the accuracy of the grasping detection, and selecting an alternative reasoning model;

[0013] S6. Input the test set image into each candidate reasoning model, compare the classification accuracy and grasping detection accuracy output by each candidate reasoning model, and select the model output with the best performance as the detection result.

[0014] The S1 dataset adds category labels of the objects to be detected based on the public dataset, expands the number of samples of the same category of objects to be detected and grasped, and labels the expanded samples.

[0015] The ECBAFormer (Efficient Convolutional Block Attention Transformer) deep learning model network for detecting the grasping target category and grasping posture information includes a feature extraction backbone network MetaFormer and a grasping detection posture generation module SGG-CNN, wherein an ECBAM module is embedded at the position of the fusion information unit in the MetaFormer.

[0016] The feature extraction backbone network MetaFormer, whose ability to extract features lies in the architecture of the network itself, includes four stages, all of which use downsampling doubling: the first stage includes four MetaFormer Block blocks, the second stage includes four MetaFormer Block blocks, the third stage includes twelve MetaFormer Block blocks, and the fourth stage includes four MetaFormer Block blocks.

[0017] Furthermore, the MetaFormer Block includes two residual sub-blocks, the first residual sub-block includes a fusion information unit for connecting information between tokens; the second residual sub-block includes a nonlinearly activated two-layer multi-layer perceptron for extracting and outputting feature information.

[0018] Furthermore, the grasping detection posture generation module SGG-CNN includes a dimension transformation convolution layer and a transposed convolution layer, and outputs the probability of the center of the grasping box, the width of the grasping box, the sine value of the grasping angle, and the cosine value of the grasping angle.

[0019] The ECBAM module sequentially obtains attention maps in the channel and spatial dimensions, and multiplies the attention maps by the input feature maps for adaptive feature training. For the channel attention map, the ECBAM module adaptively captures the dependencies between channels through one-dimensional convolution to obtain the channel attention map. For the spatial attention map, the ECBAM module generates a feature descriptor by applying average pooling and maximum pooling operations along the channel axis and connecting the channels, and generates the spatial attention map using a convolutional layer on the descriptor.

[0020] According to a second aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the method described above is implemented when the processor executes the program.

[0021] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method described above when executed by a processor.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] (1) Improve detection efficiency and accuracy: By simplifying the feature extraction network, we obtain an efficient and fast feature extraction network architecture MetaFormer, which is lightweight while ensuring the network's feature extraction capability. The network can also filter out the target background, thereby improving detection efficiency and accuracy.

[0024] (2) Increase the scope of application and improve accuracy: The robot is designed to automatically detect target objects of different categories and shapes, which improves the generalization ability of detection.

[0025] (3) Improve the success rate of robotic grasping: Through accurate detection of target category and grasping posture, the robustness of robotic grasping under different targets and environments is enhanced, and the grasping success rate is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of the intelligent grasping detection method of a robotic arm based on deep learning;

[0027] Figure 2 This is the structural diagram of the deep learning model ECBAFormer;

[0028] Figure 3 This is the MetaFormer backbone network model architecture diagram;

[0029] Figure 4 This is a schematic diagram of the combination of the channel attention module Channel Attention and the spatial attention module Spatial Attention;

[0030] Figure 5 This is the ECBAM module structure diagram. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] like Figure 1 As shown, a robot arm intelligent grasping detection method based on deep learning includes the following steps:

[0033] S1. Take the object to be detected, the grab box label and the target category label as training input, create the object grab detection and classification dataset, and divide it into training set and test set;

[0034] S2. Input the training data set into the ECBAFormer deep learning model network for training. The backbone network extracts features from the input image to generate feature maps and target categories at each stage.

[0035] S3, input the generated feature map into the SGG-CNN block, and output the center probability, width, sine and cosine values ​​of the grabbing angle of each pixel through the convolution and transposed convolution layers;

[0036] S4, processing the output result, performing non-maximum suppression on the target category information, and determining the category of the predicted target; selecting five points with the highest probability in the grasping posture information as the center of the grasping box;

[0037] S5, comparing the processed data with the label, calculating the accuracy of the grasping detection, and selecting an alternative reasoning model;

[0038] S6. Input the test set image into each candidate reasoning model, compare the classification accuracy and grasping detection accuracy output by each candidate reasoning model, and select the model output with the best performance as the detection result.

[0039] The data set S1 of this embodiment adds category labels of objects to be detected based on the public data set Cornell, expands the number of samples of objects to be detected and grasped of the same category, and labels the expanded samples.

[0040] The input image is first converted into a feature vector through the Patch Embed layer to provide input for subsequent modules. Figure 2 As shown, the ECBAForm deep learning model network for detecting grasping target categories and grasping posture information includes a feature extraction backbone network MetaFormer and a grasping detection posture generation module SGG-CNN, wherein the ECBAM module is embedded in the position of the fusion information unit in MetaFormer.

[0041] like Figure 3 As shown in the figure, the feature extraction backbone network MetaFormer has no specified fusion information unit, and the ability to extract features lies in the architecture of the network itself. The input image is first converted into a feature vector through the block embedding layer to provide input for subsequent modules. The backbone network consists of four stages, all of which use downsampling doubling: the first stage includes four MetaFormerBlock blocks, the second stage includes four MetaFormer Block blocks, the third stage includes twelve MetaFormerBlock blocks, and the fourth stage includes four MetaFormer Block blocks.

[0042] The MetaFormer Block includes two incomplete sub-modules. The first incomplete sub-module includes a fusion information unit for connecting the information between tokens; the second incomplete sub-module includes a two-layer multi-layer perceptron with nonlinear activation for extracting and outputting feature information.

[0043] The grasp detection pose generation module SGG-CNN includes a dimension transformation convolution layer and a transposed convolution layer, which outputs the probability of the center of the grasp box, the width of the grasp box, the sine value of the grasp angle, and the cosine value of the grasp angle.

[0044] like Figure 4 As shown, the ECBAM module sequentially obtains attention maps in the channel and spatial dimensions, and multiplies the attention maps by the input feature maps to perform adaptive feature training and more comprehensively capture the key information in the features. Through the gradual application of channel attention and spatial attention, ECBAM is able to adaptively adjust the importance of each channel and spatial position in the input feature map. This adaptive recalibration mechanism allows the model to dynamically focus on the most useful features based on task requirements and content context, thereby improving the model's representational capabilities and decision accuracy. For the channel attention map, the ECBAM module adaptively captures the dependencies between channels through one-dimensional convolution to obtain the channel attention map. For the spatial attention map, the ECBAM module generates a feature descriptor by applying average pooling and maximum pooling operations along the channel axis and connecting the channels, and uses a convolutional layer on the descriptor to generate the spatial attention map. As shown Figure 5 Shown is the ECBAM module structure diagram.

[0045] In actual scenarios, the targets of the object categories included in the dataset are placed on the test bench, and the ECBAFormer model is used as the visual driver to detect the detection accuracy of the model in actual scenarios.

[0046] As shown in Table 1, the method disclosed in this embodiment is compared with other commonly used methods in terms of only detecting the grasping posture and simultaneously detecting the grasping posture and the target category. It can be found that this embodiment has a higher grasping success rate than the existing methods and can also accurately identify the target category.

[0047] Table 1 Comparison of this embodiment with other experimental models on grasping posture and target category detection

[0048]

[0049]

[0050] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0051] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0052] A plurality of components in the device are connected to the I / O interface, including: an input unit, such as a keyboard, a mouse, etc.; an output unit, such as various types of displays, speakers, etc.; a storage unit, such as a disk, an optical disk, etc.; and a communication unit, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit allows the device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks. The processing unit performs the various methods and processes described above, such as the method of the present invention. For example, in some embodiments, the method of the present invention may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of the method of the present invention described above may be performed. Alternatively, in other embodiments, the CPU may be configured to perform the method of the present invention in any other appropriate manner (e.g., by means of firmware).

[0053] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0054] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0055] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0056] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A robot arm intelligent grasping detection method based on deep learning, characterized in that: The steps include: S1. Take the object to be detected, the grab box label and the target category label as training input, create the object grab detection and classification dataset, and divide it into training set and test set; S2. Input the training data set into the ECBAFormer deep learning model network for training. The backbone network extracts features from the input image to generate feature maps and target categories at each stage. S3, input the generated feature map into the SGG-CNN block, and output the center probability, width, sine and cosine values ​​of the grabbing angle of each pixel through the convolution and transposed convolution layers; S4, processing the output result, performing non-maximum suppression on the target category information, and determining the category of the predicted target; selecting five points with the highest probability in the grasping posture information as the center of the grasping box; S5, comparing the processed data with the label, calculating the accuracy of the grasping detection, and selecting an alternative reasoning model; S6, inputting the test set image into each candidate reasoning model, comparing the classification accuracy and grasping detection accuracy output by each candidate reasoning model, and selecting the model output with the best performance as the detection result; The ECBAForme deep learning model network for detecting the grasping target category and grasping posture information includes a feature extraction backbone network MetaFormer and a grasping detection posture generation module SGG-CNN, wherein an ECBAM module is embedded at the position of the fusion information unit in MetaFormer.

2. According to the deep learning-based intelligent grasping detection method for a robotic arm according to claim 1, it is characterized in that: The feature extraction backbone network MetaFormer includes four stages, all of which use downsampling doubling; The first stage includes four MetaFormer Block blocks, the second stage includes four MetaFormer Block blocks, the third stage includes twelve MetaFormer Block blocks, and the fourth stage includes four MetaFormer Block blocks.

3. According to the deep learning-based intelligent grasping detection method of a robotic arm according to claim 2, it is characterized in that: The MetaFormer Block includes two incomplete sub-modules. The first incomplete sub-module includes a fusion information unit for connecting information between tokens; the second incomplete sub-module includes a nonlinearly activated double-layer multi-layer perceptron for extracting and outputting feature information.

4. According to the deep learning-based intelligent grasping detection method for a robotic arm according to claim 1, it is characterized in that: The grasp detection posture generation module SGG-CNN includes a dimension transformation convolution layer and a transposed convolution layer, and outputs the probability of the center of the grasp box, the width of the grasp box, the sine value of the grasp angle, and the cosine value of the grasp angle.

5. According to the deep learning-based intelligent grasping detection method for a robotic arm according to claim 1, it is characterized in that: The ECBAM module sequentially obtains attention maps in channel and spatial dimensions, and multiplies the attention maps by the input feature maps for adaptive feature training.

6. The method for intelligent grasping detection of a robot arm based on deep learning according to claim 5, characterized in that: The ECBAM module adaptively captures the dependencies between channels through one-dimensional convolution, thereby obtaining the channel attention map.

7. The method for intelligent grasping detection of a robot arm based on deep learning according to claim 5, characterized in that: The ECBAM module generates a feature descriptor by applying average pooling and maximum pooling operations along the channel axis and connecting the channels, and a convolutional layer is used on the descriptor to generate the spatial attention map.

8. The method for intelligent grasping detection of a robotic arm based on deep learning according to claim 1, characterized in that: The S1 dataset adds category labels of the objects to be detected based on the public dataset, expands the number of samples of the same category of objects to be detected and grasped, and labels the expanded samples.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images