Pixel-level grasping pose detection method and system based on global and local information

Abstract

The present invention discloses a pixel-level grasping posture detection method and system based on global and local information; the method comprises: acquiring a scene image and preprocessing the scene image; inputting the preprocessed scene image into a trained pixel-level grasping posture detection network, outputting the plane grasping posture of each pixel point, mapping the plane grasping posture to a real scene through a posture conversion relationship, and grasping the target object by a robotic arm; wherein the trained pixel-level grasping posture detection network has the following working principles: extracting features from the scene image to obtain image features; performing denoising on the extracted image features; performing feature fusion on the denoised image features to obtain a feature map containing different receptive fields; performing classification prediction on the feature map containing different receptive fields, and outputting the plane grasping posture of each pixel point.

Description

Technical Field

[0001] The present invention relates to the technical field of grasping posture detection, and in particular to a pixel-level grasping posture detection method and system based on global and local information. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] Grasping pose detection is a fundamental technology for robotic object manipulation. With the development of intelligent robots, precise manipulation requires increasingly high accuracy in grasping pose detection, and pose detection algorithms must be applicable to complex scenes with multiple objects. Current grasping detection algorithms mostly use convolutional neural networks to predict possible grasping poses based on local information between objects. These algorithms rarely incorporate global information between different objects in the scene to predict grasping poses in multi-object scenes.

[0004] In their paper "High-performance Pixel-level Grasp Detection based on Adaptive Grasping and Grasp-aware Network," published in IEEE Transactions on Industrial Electronics, Dexin Wang et al. proposed converting traditional rectangular box-based grasping pose detection into pixel-by-pixel directed arrow predictions. This improves the density of grasping detection and avoids missing possible grasping poses. They also use dilated convolution and feature fusion to acquire features at different scales, enhancing the model's detection and generalization capabilities. However, this method is still based on traditional convolutional neural networks. While dilated convolution and feature fusion improve the ability to perceive and extract global information, dilated convolution lacks particularly strong global information aggregation capabilities, and therefore does not perform well in complex multi-object scenes. Relying solely on local information cannot effectively address the interactions between objects in complex multi-object scenes.

[0005] At present, the processing and application of global information are mainly based on the Transformer mechanism, which obtains the correlation between different patches of the image and has achieved good results in many fields of image processing. In the paper "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" published by ZeLiu et al. at the 2021 ICCV conference, a Transformer model with cyclic shift and local window attention mechanism was proposed, which can not only obtain global information and obtain feature maps at different levels, but also reduce the amount of calculation. This method was later applied to target detection, semantic segmentation and other fields, and achieved very good results. Although the Swin Transformer has a strong ability to extract global information, local information may be destroyed in the process of feature extraction, and pixel-by-pixel prediction requires very detailed information. Therefore, the simple Swin Transformer is not effective for capturing pose prediction. It needs to be improved to improve its local information processing capabilities, and it must be able to integrate local information and global information well in the future.

[0006] In the process of implementing the present invention, the inventors found that the following technical problems exist in the prior art: the conventional grasping detection algorithm uses a convolutional neural network to obtain local information of objects in the scene, and lacks attention to global information. This leads to inaccurate grasping detection in complex scenes with multiple objects, limiting the accuracy and usability of the grasping algorithm. Summary of the Invention

[0007] In order to solve the problem of low prediction accuracy in complex scenes with multiple objects in the existing technology, the present invention provides a pixel-level grasping pose detection method and system based on global and local information; it proposes an improved FFN module to improve the local information extraction and aggregation capability of Transformer, and then designs an adaptive feature fusion module to fuse different features. Finally, different grasping parameters are predicted through three parallel network branches, and the parameters are fused to obtain the grasping configuration, thereby obtaining reliable grasping and improving the accuracy and generalization of grasping pose detection.

[0008] In a first aspect, the present invention provides a pixel-level grasping pose detection method based on global and local information;

[0009] Pixel-level grasp pose detection method based on global and local information, including:

[0010] Acquire a scene image and preprocess the scene image;

[0011] The pre-processed scene image is input into the trained pixel-level grasping pose detection network, which outputs the plane grasping pose of each pixel. After the pose conversion relationship, the plane grasping pose is mapped to the real scene, and the robotic arm grasps the target object.

[0012] Among them, the working principle of the trained pixel-level grasping pose detection network includes: extracting features from the scene image to obtain image features; performing denoising on the extracted image features; performing feature fusion on the denoised image features to obtain feature maps containing different receptive fields; performing classification prediction on the feature maps containing different receptive fields, and outputting the planar grasping pose of each pixel point.

[0013] In a second aspect, the present invention provides a pixel-level grasping pose detection system based on global and local information;

[0014] Pixel-level grasp pose detection system based on global and local information, including:

[0015] An acquisition module is configured to: acquire a scene image and preprocess the scene image;

[0016] The pose detection module is configured to: input the preprocessed scene image into the trained pixel-level grasp pose detection network, output the plane grasp pose of each pixel point, and map the plane grasp pose to the real scene through the pose conversion relationship, so that the robot arm can grasp the target object;

[0017] Among them, the working principle of the trained pixel-level grasping pose detection network includes: extracting features from the scene image to obtain image features; performing denoising on the extracted image features; performing feature fusion on the denoised image features to obtain feature maps containing different receptive fields; performing classification prediction on the feature maps containing different receptive fields, and outputting the planar grasping pose of each pixel point.

[0018] In a third aspect, the present invention further provides an electronic device, comprising:

[0019] a memory for non-transitory storage of computer-readable instructions; and

[0020] a processor for executing said computer-readable instructions,

[0021] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

[0022] In a fourth aspect, the present invention further provides a storage medium that non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.

[0023] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, wherein the computer program is used to implement the method described in the first aspect when running on one or more processors.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] By fusing global and local information, and leveraging the global information extraction capabilities of Transformer and the local information aggregation capabilities of convolutional neural networks, the two methods are integrated to achieve complementarity, detecting possible grasping configurations for each pixel in the input image, and then filtering to obtain reliable planar grasping configurations for objects in the image.

[0026] It is proposed to improve the FFN structure and introduce the convolutional network into the Swin Transformer to enhance the network's ability to extract and aggregate local information. Then, an adaptive feature fusion module is proposed to adaptively learn to fuse different features to obtain a sufficiently fine feature map, thereby improving the accuracy and generalization ability of pixel-level prediction.

[0027] The present invention adopts a Transformer-based feature network to deeply explore the relationship between different objects and extract deep features that contain the interactive relationship between multiple objects.

[0028] A local information extraction module is added to the Transformer to enhance the ability to extract local information of objects; an adaptive feature fusion module is used to learn the importance of features in grasping pose prediction, increase the weight of important information in the prediction, and enhance the prediction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0030] Figure 1 This is a flowchart of a pixel-level grasping pose detection method based on the fusion of global information and local information according to the first embodiment of the present invention;

[0031] FIG2( a ) is a Swin Transformer network structure according to the first embodiment of the present invention;

[0032] FIG2( b ) is the internal network structure of the improved Swin Transformer Block according to the first embodiment of the present invention;

[0033] FIG3( a ) is a diagram showing the structure of a pixel-level grasping pose detection network based on the fusion of global and local information according to the first embodiment of the present invention;

[0034] FIG3( b ) is a diagram showing the internal structure of a feature fusion network according to the first embodiment of the present invention;

[0035] Figure 4(a)-Figure 4(p) This is an example of pose prediction results captured from the PLGP dataset of Example 1 of the present invention. DETAILED DESCRIPTION

[0036] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0037] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0039] All data in this embodiment is obtained in compliance with laws and regulations and based on the consent of the user, and is used legally.

[0040] Pixel-level grasping pose detection refers to predicting the possible planar grasping configuration at each pixel point of the input RGB image or depth image pixel by pixel, mapping it to the real scene through subsequent pose transformation relationship, and then achieving reliable grasping through a robotic arm. It plays an important role in computer vision, human-computer interaction, robotic operation and other fields.

[0041] Example 1

[0042] This embodiment provides a pixel-level grasping pose detection method based on global and local information;

[0043] like Figure 1 As shown in Figure 2, the pixel-level grasp pose detection method based on global and local information includes:

[0044] S101: Acquire a scene image and preprocess the scene image;

[0045] S102: Input the pre-processed scene image into the trained pixel-level grasping pose detection network, output the plane grasping pose of each pixel point, and map the plane grasping pose to the real scene through the pose conversion relationship, so that the robot arm can grasp the target object;

[0046] The working principle of the trained pixel-level grasping pose detection network includes the following:

[0047] Extract features from the scene image to obtain image features;

[0048] Perform noise reduction on the extracted image features;

[0049] Perform feature fusion on the image features after denoising to obtain feature maps with different receptive fields;

[0050] Classify and predict the feature maps containing different receptive fields, and output the plane grasping pose of each pixel.

[0051] Furthermore, the scene image may be an RGB image, a depth image or an RGB-D image.

[0052] Furthermore, the preprocessing of the scene image includes:

[0053] Perform scale scaling, angle rotation, Gaussian noise addition, and translation operations on the image in sequence;

[0054] For the image after the translation operation, a center cropping method is used to crop an image of a set size, and the cropped image is normalized.

[0055] Exemplarily, the preprocessing of the scene image includes:

[0056] Random scaling is performed within the scale of [1, 1.2], random rotation is performed within the range of [-30°, 30°], Gaussian noise is added, and then the center cropping method with a maximum horizontal and vertical translation of 30 pixels is used to obtain a 320×320 input image, and finally the image is normalized.

[0057] Input=Normalize(Crop(Gaussian(Rotate(Rescale(I))))) (1)

[0058] Among them, Crop refers to center cropping, Gaussian operation refers to adding random Gaussian noise, Rotate refers to random rotation operation, and Rescale is random scaling operation.

[0059] Furthermore, as shown in FIG3( a ), the trained pixel-level grasping pose detection network has a network structure including:

[0060] The backbone network, denoising network, feature fusion network and decoder are connected in sequence.

[0061] Furthermore, the backbone network is implemented using an improved Swin Transformer network.

[0062] Furthermore, as shown in FIG2( a ), the improved Swin Transformer network includes:

[0063] The block partition Patch Partition layer, the first-order module Stage1, the second-order module Stage2, the third-order module Stage3 and the fourth-order module Stage4 are connected in sequence;

[0064] The first-stage module Stage1 includes: a linear coding Linear Embedding layer, a first improved Swin Transformer Block, and a second improved Swin Transformer Block connected in sequence;

[0065] The second-stage module Stage2 includes: a first Patch Merging layer, a third improved Swin Transformer Block, and a fourth improved Swin Transformer Block connected in sequence;

[0066] The third-order module Stage3 includes: a second Patch Merging layer, a fifth improved Swin Transformer Block, a sixth improved Swin Transformer Block, a seventh improved Swin Transformer Block, an eighth improved Swin Transformer Block, a ninth improved Swin Transformer Block, and a tenth improved Swin Transformer Block connected in sequence;

[0067] The fourth-stage module Stage4 includes: a third Patch Merging layer, an eleventh improved Swin Transformer Block and a twelfth improved Swin Transformer Block connected in sequence.

[0068] Furthermore, the internal structures of the first improved Swin Transformer Block, the second improved Swin Transformer Block, the third improved Swin Transformer Block, the fourth improved Swin Transformer Block, the fifth improved Swin Transformer Block, the sixth improved Swin Transformer Block, the seventh improved Swin Transformer Block, the eighth improved Swin Transformer Block, the ninth improved Swin Transformer Block, the tenth improved Swin Transformer Block, the eleventh improved Swin Transformer Block and the twelfth improved Swin Transformer Block are consistent.

[0069] Furthermore, the improved Swin Transformer network is obtained by replacing all Swin Transformer Blocks of the Swin Transformer network with the first improved Swin Transformer Block.

[0070] Furthermore, as shown in FIG2( b ), the first improved Swin Transformer Block includes:

[0071] A first basic unit and a second basic unit connected in sequence;

[0072] The first basic unit includes: a first normalization layer LN (LayerNormalization), a window multi-head self-attention mechanism layer (Window Multi-head Self-Attention, W-MSA), an adder J1, a second normalization layer LN, a first improved FFN module and an adder J2 connected in sequence; wherein the input end of the adder J1 is residually connected to the input end of the first normalization layer LN; the input end of the adder J2 is residually connected to the input end of the second normalization layer LN;

[0073] The second basic unit includes: a third normalization layer LN, a shifted window multi-head self-attention mechanism layer (Shifted Window Multi-head Self-Attention, SW-MSA), an adder J3, a fourth normalization layer LN, a second improved FFN module and an adder J4 connected in sequence; wherein the input end of the adder J3 is residually connected to the input end of the third normalization layer LN; the input end of the adder J4 is residually connected to the input end of the fourth normalization layer LN;

[0074] The input end of the first normalization layer LN serves as the input end of the first improved Swin Transformer Block; the output end of the adder J4 serves as the output end of the first improved Swin Transformer Block; the output end of the adder J2 is connected to the input end of the third normalization layer LN.

[0075] It should be understood that the Window Multi-head Self-Attention mechanism layer (W-MSA) is an improvement to the traditional multi-head self-attention mechanism. It divides the input image into small windows and then performs attention calculations on the windows to reduce computational complexity.

[0076] It should be understood that the Shifted Window Multi-head Self-Attention (SW-MSA) mechanism is designed to address the problem that window multi-head self-attention can only perceive information within the window. It shifts the input feature image leftward and upward by half the window size, and then moves the feature information that has been moved out of the window in the upper left corner to the lower right corner. This allows the window to perceive information at different positions, enabling information exchange between windows. The Shifted Window Multi-head Self-Attention mechanism shifts the elements of the feature map so that non-adjacent elements are adjacent. Then, through window attention calculation, the correlation between non-adjacent elements is obtained.

[0077] Furthermore, the internal structure of the first improved FFN module is consistent with that of the second improved FFN;

[0078] The first improved FFN module includes:

[0079] The 1*1 convolution layer C1, the dimension transformation Reshape layer R1, the 3*3 depth-wise separable convolution layer C2, the dimension transformation Reshape layer R2 and the convolution layer C3 are connected in sequence.

[0080] Among them, the convolutional neural network (convolutional layer C1, depth-wise separable convolutional layer C2 and convolutional layer C3) extracts local information, and the improved Swin Transformer network extracts global information.

[0081] Furthermore, the working principle of the first improved FFN module includes:

[0082] After processing through the sliding window attention mechanism layer, the feature code P is obtained, and the channel correlation of the feature code P is mined using one-dimensional convolution to obtain the feature code L;

[0083] Then the feature code L is adjusted from one dimension to two-dimensional image feature F, and sent to depth-wise convolution to extract local features F′;

[0084] The obtained F′ is then adjusted to a one-dimensional feature code L′, and the one-dimensional convolution process is continued to obtain the feature map P′.

[0085] Then it is sent to the subsequent Swin Transformer Block for processing.

[0086] L=Conv1D(P) (3)

[0087] F′=DWConv2D(Reshape(L)) (4)

[0088] P′=Conv1D(Reshape(F′)) (5)

[0089] After multiple cycles, the obtained feature map contains relatively rich information, which facilitates pixel-by-pixel prediction of the image.

[0090] Furthermore, the working principle of the improved Swin Transformer network includes:

[0091] The input image is cut into patches and then embedded into vectors to obtain the token of the input image. The image is then passed through four Swin Transformer Blocks for feature extraction to obtain a feature map.

[0092] Furthermore, the working principle of the first improved Swin Transformer Block includes:

[0093] Step 102-a1: Encoding S of the input image Token i , a query vector Q is calculated using a window multi-head self-attention mechanism layer or a linear transformation layer of a shifted window multi-head self-attention mechanism layer i ,key vector K iand a value vector V i ; Then Q j and K i After the dot product, we get a score to represent the input S j With S i and use this score as a weight with V i After weighted summation, we get the attention feature head i This process is called scaled dot-product attention and is expressed in matrix form as:

[0094]

[0095] Where d is the dimension of Q and K, and B is the relative position encoding;

[0096] Step 102-a2: resize the extracted attention feature sequence into a two-dimensional feature image;

[0097] Step 102-a3: performing a convolution operation on the obtained two-dimensional feature image to extract local information and obtain the relationship between different feature points and their surrounding feature points;

[0098] Step 102-a4: Resize the convolution-processed feature map into a one-dimensional feature sequence and send it to the subsequent Swin Transformer Block for further calculation.

[0099] Swin Transformer is based on cyclic shift and local window self-attention, and its basic structure is shown in Figure 2(b).

[0100] The window self-attention mechanism layer inside the Swin Transformer Block limits the attention calculation to each window to reduce the amount of computation.

[0101] In order to obtain better performance and deeply extract local information of the image, the present invention designs an improved FFN module to replace the FFN module of Swin Transformer, introduces a convolutional neural network, and enhances the local information extraction capability of the model.

[0102] Furthermore, as shown in FIG3( a ), the denoising network has a network structure including:

[0103] Adder J5, the input end of the adder J5 is respectively connected to the output end of the first-order module Stage1, the second-order module Stage2, the third-order module Stage3 and the fourth-order module Stage4;

[0104] The output end of the adder J5 is connected to the input end of the Refine layer;

[0105] The output of the Refine layer is connected to the input of the adder J6, and the input of the adder J6 is also connected to the input of the first-order module Stage1; the output of the adder J6 is respectively connected to the input of the four layers of the Feature Pyramid Network (FPN) network from top to bottom, the output of the four layers of the FPN network from top to bottom is connected to the input of the splicing unit, and the output of the splicing unit is connected to the input of the feature fusion network.

[0106] Furthermore, the working principle of the denoising network includes:

[0107] The four-layer feature maps obtained by the improved Swin Transformer network are marked as S1, S2, S3, and S4, which are the results of Stage 1, Stage 2, Stage 3, and Stage 4, respectively, and their sizes gradually decrease.

[0108] Pooling is used to reduce the size of S1 and S2 to the size of S3. Bilinear interpolation is used to enlarge the size of S4 to the size of S3. The processed feature maps are added together, and then refined and balanced by the Refine layer to obtain feature map A3.

[0109] Use bilinear interpolation to restore feature map A3 to the original size of S1, and then add it to S1 to obtain feature map A1;

[0110] Use bilinear interpolation to restore feature map A3 to the original size of S2, and then add it to S2 to obtain feature map A2;

[0111] Use the pooling operation on the feature map A3 to restore it to the original size of S4, and then add it to S4 to obtain the feature map A4;

[0112] Then, A4 is bilinearly interpolated to the size of A3 and then added to A3 to obtain M3;

[0113] Scale M3 bilinearly to the size of A2 and then add it to A2 to get M2.

[0114] Then, M2 is bilinearly interpolated to the size of A1 and then added to A1 to obtain M1.

[0115] Then use bilinear interpolation to enlarge M2, M3, and A4 to the size of A1;

[0116] Then concatenate M1, M2, M3, and A4 into a high-dimensional feature map;

[0117] Then the convolutional layer is used to reduce the dimension and obtain the input feature map of the feature fusion network.

[0118] The Refine layer is to denoise and further refine the features.

[0119] Furthermore, as shown in FIG3( b ), the feature fusion network has a network structure including:

[0120] Parallel atrous convolution layer C4, atrous convolution layer C5, atrous convolution layer C6 and atrous convolution layer C7;

[0121] The input ends of the dilated convolution layer C4, dilated convolution layer C5, dilated convolution layer C6, and dilated convolution layer C7 are all connected to the output end of the denoising network;

[0122] The outputs of the dilated convolutional layer C4, the dilated convolutional layer C5, the dilated convolutional layer C6, and the dilated convolutional layer C7 are all connected to the input of the convolutional layer C8.

[0123] The output of the convolutional layer C8 is connected to the input of the Sigmoid function layer;

[0124] The output of the Sigmoid function layer is connected to the input of the Split segmentation function layer;

[0125] The output end of the Split segmentation function layer outputs four independent weight parameters. Based on the four independent weight parameters, the output data of the dilated convolution layer C4, dilated convolution layer C5, dilated convolution layer C6 and dilated convolution layer C7 are weighted, and then the weighted data are summed using the adder;

[0126] The summed data is concatenated with the output data of the first-order module Stage1, and the concatenated result is input into the convolutional layer C9. The output value of the convolutional layer C9 is the output value of the feature fusion network.

[0127] It should be understood that since it is very difficult to predict the capture configuration for each pixel of the input, the input feature map needs to contain sufficiently fine and rich features. Therefore, the present invention designs adaptive feature fusion to enhance the information richness of the feature map in the decoding stage.

[0128] Furthermore, the working principle of the feature fusion network is as follows:

[0129] Use four dilation coefficients of dilated convolution to process the feature map after balanced feature pyramid (BFP) denoising and FPN processing to expand the receptive field of the decoding stage, and obtain four feature maps f1 f2 f3 f4 with different receptive fields;

[0130] The convolutional network is then used to calculate the importance of these four feature maps for subsequent decoding, and we get Then W is divided into 4 independent weight parameters, which are multiplied pixel by pixel with the feature maps f1 f2 f3 f4 to amplify important feature elements and suppress unimportant features.

[0131] f i =Conv i (F) (6)

[0132] W=Sigmoid(Conv fw (Concat(f1,f2,f3,f4))) (7)

[0133]

[0134] Among them, Conv i It is a dilated convolution, Conv fw is the convolution module that calculates the importance weight, W i is the weight after dividing W, and ⊙ is the pixel-by-pixel multiplication.

[0135] The feature map output by the first-order module Stage1 of the backbone network is introduced into the fused F L Among the features;

[0136] F AF =Conv AF (Concat(F L ,Conv b1 (F b1 ))) (9)

[0137] Among them, Conv b1 It is a dimensionality reduction operation.

[0138] Furthermore, the network structure of the decoder includes:

[0139] Three parallel branches: the first branch, the second branch and the third branch;

[0140] The first branch includes the convolutional layer C10, the deconvolutional layer F1 and the upsampling layer U1 connected in sequence;

[0141] The second branch includes the convolutional layer C11, the deconvolution layer F2 and the upsampling layer U2 connected in sequence;

[0142] The third branch includes the convolutional layer C12, the deconvolution layer F3 and the upsampling layer U3 connected in sequence;

[0143] The input ends of the convolutional layer C10, the convolutional layer C11 and the convolutional layer C12 are all connected to the output end of the convolutional layer C9.

[0144] Furthermore, the working principle of the decoder includes:

[0145] The fused feature F AF The images are fed into three parallel branches respectively, and each branch passes through the convolution layer, deconvolution layer and upsampling layer to obtain the final planar grasping pose; the planar grasping pose includes: grasping confidence, angle classification and grasping width.

[0146] Furthermore, the training process of the trained pixel-level grasping pose detection network includes:

[0147] Constructing a training set and a test set; the training set and the test set are both scene images with known planar grasping poses of each pixel;

[0148] The training set is input into the pixel-level grasping pose detection network to train the network. When the loss function value of the network no longer decreases, the training is stopped to obtain the pixel-level grasping pose detection network after preliminary training.

[0149] The test set is then input into the pixel-level grasping pose detection network to test the pixel-level grasping pose detection network. When the test accuracy exceeds the set threshold, the training is stopped to obtain the trained pixel-level grasping pose detection network.

[0150] Furthermore, the training set and test set are the PLGP dataset, which is a dense grasping annotation dataset generated through a virtual simulation environment. It contains 45,550 RGB-D images from different perspectives in cluttered scenes. Each image has dense annotations, and a total of more than 58 million grasping poses.

[0151] The PLGP dataset contains RGB images and depth images. You can choose to use RGB images, depth images, or RGB-D images as input. The input image is labeled I, where I can be an RGB image, a depth image, or an RGB-D image.

[0152] Furthermore, the loss function adopts the Focal Loss loss function, which can balance the imbalance of positive and negative samples in the data set.

[0153] Use Focal Loss function to optimize network parameters:

[0154]

[0155]

[0156] Among them, P is the output result of size K×H×W, P *is the training target, γ and α are the parameters of Focal Loss, 1() is the judgment function, and the value is 1 if the expression in the brackets is satisfied, otherwise it is 0.

[0157] Because the output is three parallel branch networks, the total loss function is:

[0158]

[0159] β1, β2, β3 are the weights of each part, which are set to 100, 1000 and 2000 respectively.

[0160] The present invention uses 8 images as a batch input, the backbone network does not use pre-trained weight parameters, and all network layers are initialized using kaiming. The optimizer selects AdamW optimizer, and the initial learning rate is 5×10 -7 , using the Warmup technique, after one round of training, the basic learning rate is 5×10 -5 , weight decay is set to 0.05. The parameters are optimized using formula (12) as the loss function. The entire network is trained for a total of 100 epochs on a single GPU.

[0161] Furthermore, the feature extraction of the scene image to obtain image features specifically includes:

[0162] The Patch Partition layer outputs the feature map after image segmentation encoding, which passes through the first-order module Stage1, the second-order module Stage2, the third-order module Stage3 and the fourth-order module Stage4 in sequence, all of which output image features. The first-order module Stage1 outputs global information.

[0163] The image features obtained by the four modules are subjected to BFP denoising and FPN fusion to obtain the denoised feature map.

[0164] Furthermore, the feature fusion of the image features after the noise reduction process is performed to obtain feature maps containing different receptive fields, specifically including:

[0165] Step 102-b1: Use four different dilation coefficients of the dilated convolution to obtain four feature images with different receptive fields;

[0166] Step 102-b2: Concatenate the feature images of the four different receptive fields and use a convolutional network to learn their importance for subsequent predictions to obtain importance parameters, normalize the importance parameters, and map the importance parameters to the range (0, 1);

[0167] Step 102-b3: Split the importance parameter tensor into four parts, and multiply them pixel by pixel with the feature maps of four different receptive fields to obtain weighted feature maps. The features of important parts are amplified and the features of unimportant parts are suppressed. The feature maps are then concatenated and dimensionality reduced.

[0168] Step 102-b4: Reduce the dimension of the features of the first layer in the backbone network and concatenate and reduce the dimension of the feature map obtained in step 102-b3 to obtain a feature map containing different receptive fields.

[0169] Furthermore, the classification prediction of the feature maps containing different receptive fields and the output of the plane grasping pose representation of each pixel point specifically include:

[0170] The planar grasping pose representation of each pixel point refers to: the grasping confidence, grasping angle and grasping width of each pixel point.

[0171] The pixel points with the highest confidence are selected based on the grasping confidence. Then, through coordinate system transformation, the pixel coordinate system is converted to the camera coordinate system and then to the world coordinate system to obtain the position of the grasped point in the three-dimensional world. The robotic arm is moved to the specified position and rotated according to the grasping angle. Then, the gripper of the robotic arm is opened according to the grasping width to grasp the object and complete the grasping operation.

[0172] The grasping posture representation is divided into the spatial coordinates (three-dimensional) of the grasping point, the grasping direction and the width of the robotic claw opening. Since plane grasping is from top to bottom, the grasping direction is fixed, so the grasping direction degenerates into the angle of rotation of the robotic arm around the normal of the table plane.

[0173] It should be understood that the above technical solution provides a method for extracting local and global information from the input image in scenes with multiple objects using an improved SwinTransformer network, and then performing feature fusion through a subsequent feature fusion module. The entire process includes an improved SwinTransformer network, an improved FFN module, local and global information extraction, adaptive feature fusion, and finally the use of Focal Loss to mitigate the imbalance between positive and negative samples.

[0174] The effect of the present invention can be further illustrated by the following experiments:

[0175] 1. Experimental Conditions

[0176] The hardware conditions for all verification experiments of the present invention are: Ubuntu 20.04L, CPU: Intel(R) Xeon(R) Gold 6226R CPU @ 2.90 GHz, and 1 NVIDIA GeForce 3090 graphics card.

[0177] The software environment used in the experiment is: python 3.7, pytorch = 1.7.1, torchvision = 0.8.2

[0178] The experimental data comes from the PLGP dataset. Using simulation technology, the PLGP dataset collects 4555 grasping poses of 3D models, including 45,550 RGB-D images from different scenes and viewpoints. The training and test sets are split in an 8:2 ratio.

[0179] 2. Experimental content and results

[0180] The present invention conducted an experiment on object grasping position prediction in a complex scene with multiple objects on the PLGP dataset. The evaluation indicators used in the present invention mainly include Success Rate (SR), Percent Cleared (PC), and Average Precision (AP), which are described in detail as follows:

[0181] SR: Objects in the scene are grasped one by one until (a) no objects remain or (b) the robot fails to grasp them 5 times in a row. The success rate measures the percentage of grasp attempts that lift an object. It is calculated as:

[0182]

[0183] Where T S is the number of times the object is successfully lifted, T total is the total number of attempts.

[0184] PC: The ratio of the number of objects successfully removed from the scene after the above grasping test to the total number of objects in the scene. It measures the ability to successfully remove objects. The calculation formula is:

[0185]

[0186] Among them, P S is the number of times the object is successfully moved out of the scene, P total is the total number of objects in the scene.

[0187] AP: The average precision is calculated by testing the top 50 grasp poses with the highest confidence. After each grasp, the scene is reset to its initial state to ensure that the scene is the same for each test.

[0188] Table 1 compares the proposed method with previous algorithms. The evaluation metrics used are the aforementioned SR, PC, and Ap, with larger values indicating better performance. Table 1 demonstrates the proposed method's superior performance to previous algorithms. Furthermore, ablation experiments were conducted. Tables 2, 3, and 4 demonstrate the effectiveness of the improved FFN module, adaptive feature fusion, and the Focal Loss function, respectively.

[0189] Table 1 Crawl prediction results

[0190]

[0191] Table 2 verifies the effectiveness of the improved FFN module

[0192]

[0193] Table 3 verifies the effectiveness of adaptive feature fusion

[0194]

[0195] Table 4 verifies the effectiveness of the Focal loss function

[0196]

[0197] Example 2

[0198] This embodiment provides a pixel-level grasping pose detection system based on global and local information;

[0199] Pixel-level grasp pose detection system based on global and local information, including:

[0200] An acquisition module is configured to: acquire a scene image and preprocess the scene image;

[0201] The pose detection module is configured to: input the preprocessed scene image into the trained pixel-level grasp pose detection network, output the plane grasp pose of each pixel point, and map the plane grasp pose to the real scene through the pose conversion relationship, so that the robot arm can grasp the target object;

[0202] Among them, the working principle of the trained pixel-level grasping pose detection network includes: extracting features from the scene image to obtain image features; performing denoising on the extracted image features; performing feature fusion on the denoised image features to obtain feature maps containing different receptive fields; performing classification prediction on the feature maps containing different receptive fields, and outputting the planar grasping pose of each pixel point.

[0203] It should be noted that the acquisition module and posture detection module described above correspond to steps S101 to S102 in Example 1. The examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the modules described above, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0204] The descriptions of the various embodiments in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0205] The proposed system can be implemented in other ways. For example, the system embodiment described above is merely illustrative. For example, the above module division is only a logical function division. In actual implementation, other division methods may be used. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not implemented.

[0206] Example 3

[0207] This embodiment also provides an electronic device, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the method described in the above embodiment one.

[0208] Example 4

[0209] This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in the first embodiment is performed.

[0210] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A pixel-level grasp pose detection method based on global and local information, characterized by: include: Acquire a scene image and preprocess the scene image; The pre-processed scene image is input into the trained pixel-level grasping pose detection network, which outputs the plane grasping pose of each pixel. After the pose conversion relationship, the plane grasping pose is mapped to the real scene, and the robotic arm grasps the target object. The trained pixel-level grasping pose detection network works as follows: extracting features from scene images to obtain image features; performing noise reduction on the extracted image features; performing feature fusion on the noise-reduced image features to obtain feature maps with different receptive fields; performing classification prediction on the feature maps with different receptive fields and outputting the plane grasping pose of each pixel. The trained pixel-level grasp pose detection network has the following network structure: The backbone network, denoising network, feature fusion network and decoder are connected in sequence; The backbone network is implemented using an improved Swin Transformer network; The improved Swin Transformer network is obtained by replacing all Swin Transformer Blocks of the Swin Transformer network with the first improved Swin Transformer Block; Among them, the first improved Swin Transformer Block includes: A first basic unit and a second basic unit connected in sequence; The first basic unit includes: a first normalization layer LN, a window multi-head self-attention mechanism layer, an adder J1, a second normalization layer LN, a first improved FFN module and an adder J2 connected in sequence; wherein the input end of the adder J1 is residually connected to the input end of the first normalization layer LN; the input end of the adder J2 is residually connected to the input end of the second normalization layer LN; The second basic unit includes: a third normalization layer LN, a shifted window multi-head self-attention mechanism layer, an adder J3, a fourth normalization layer LN, a second improved FFN module and an adder J4 connected in sequence; wherein the input end of the adder J3 is residually connected to the input end of the third normalization layer LN; the input end of the adder J4 is residually connected to the input end of the fourth normalization layer LN; The input end of the first normalization layer LN serves as the input end of the first improved Swin Transformer Block; the output end of the adder J4 serves as the output end of the first improved Swin Transformer Block; the output end of the adder J2 is connected to the input end of the third normalization layer LN; The internal structure of the first improved FFN module is consistent with that of the second improved FFN; The first improved FFN module includes: The 1*1 convolution layer C1, the dimension transformation Reshape layer R1, the 3*3 depth-wise separable convolution Depth-wise layer C2, the dimension transformation Reshape layer R2 and the convolution layer C3 are connected in sequence.

2. The pixel-level grasping pose detection method based on global and local information as claimed in claim 1 is characterized in that: The first improved FFN module has the following working principles: Feature encoding is obtained after processing through the sliding window attention mechanism layer , using one-dimensional convolution to mine feature encoding Channel correlation, get feature encoding ; Then encode the features Adjust from one-dimensional to two-dimensional image features , sent to Depth-wise convolution to extract local features ; Then the obtained Adjust to one-dimensional feature encoding , continue to use one-dimensional convolution to obtain the feature map .

3. The pixel-level grasping pose detection method based on global and local information as claimed in claim 1 is characterized in that: The improved Swin Transformer network works as follows: The input image is cut into patches and then embedded into vectors to obtain the input image token. Then, four Swin Transformer Blocks are used for feature extraction to obtain a feature map. The working principle of the first improved Swin Transformer Block includes: Encoding of input image token , a window multi-head self-attention mechanism layer or a linear transformation layer of a shifted window multi-head self-attention mechanism layer is used to calculate a vector , vector and a vector ; then and After the dot product, we get a score to represent the input and and use this score as a weight with After weighted summation, we get the attention feature ; This process is called scaled dot product attention and is expressed in matrix form as: (2) in, yes and Dimensions, It is a relative position encoding; Resize the extracted attention feature sequence into a two-dimensional feature image; Perform convolution operation on the obtained two-dimensional feature image to extract local information and obtain the relationship between different feature points and their surrounding feature points; The convolutional feature map is resized into a one-dimensional feature sequence and sent to the subsequent SwinTransformer Block for further calculation.

4. The pixel-level grasping pose detection method based on global and local information as claimed in claim 1 is characterized in that: The denoising network works as follows: The four-layer feature maps obtained by the improved Swin Transformer network are marked as S1, S2, S3, and S4, which are the results of Stage 1, Stage 2, Stage 3, and Stage 4, respectively, and their sizes gradually decrease; Pooling is used to reduce the size of S1 and S2 to the size of S3. Bilinear interpolation is used to enlarge the size of S4 to the size of S3. The processed feature maps are added together, and then refined and balanced by the Refine layer to obtain feature map A3. Use bilinear interpolation to restore feature map A3 to the original size of S1, and then add it to S1 to obtain feature map A1; Use bilinear interpolation to restore feature map A3 to the original size of S2, and then add it to S2 to obtain feature map A2; Use pooling operation on feature map A3 to restore it to the original size of S4, and then add it to S4 to obtain feature map A4; Then, A4 is bilinearly interpolated to the size of A3 and then added to A3 to obtain M3; Scale M3 bilinearly to the size of A2 and then add it to A2 to get M2. Then, M2 is bilinearly interpolated to the size of A1 and then added to A1 to obtain M1. Then use bilinear interpolation to enlarge M2, M3, and A4 to the size of A1; Then concatenate M1, M2, M3, and A4 into a high-dimensional feature map; Then the convolutional layer is used to reduce the dimension and obtain the input feature map of the feature fusion network.

5. The pixel-level grasping pose detection method based on global and local information as claimed in claim 1 is characterized in that: The working principle of the feature fusion network is as follows: The feature maps after balanced feature pyramid denoising and FPN processing are processed using four dilation coefficients of dilated convolution to expand the receptive field of the decoding stage and obtain four feature maps with different receptive fields. ; The convolutional network is then used to calculate the importance of these four feature maps for subsequent decoding, and we get , and then Split into 4 independent weight parameters, the weight parameters are respectively related to the feature map Multiply pixel by pixel to amplify important feature elements and suppress unimportant features; (6) (7) in, is a two-dimensional image feature, is a dilated convolution, is a convolutional module that calculates importance weights, It will The weight after segmentation, ⊙, is multiplied pixel by pixel; The feature map output by the first-order module Stage1 of the backbone network is introduced into the fused Among the features; (9) in, is the fused feature, It is a dimensionality reduction operation.

6. A pixel-level grasp pose detection system based on global and local information, characterized by: include: An acquisition module is configured to: acquire a scene image and preprocess the scene image; The pose detection module is configured to: input the preprocessed scene image into the trained pixel-level grasp pose detection network, output the plane grasp pose of each pixel point, and map the plane grasp pose to the real scene through the pose conversion relationship, so that the robot arm can grasp the target object; The trained pixel-level grasping pose detection network works as follows: extracting features from scene images to obtain image features; performing noise reduction on the extracted image features; performing feature fusion on the noise-reduced image features to obtain feature maps with different receptive fields; performing classification prediction on the feature maps with different receptive fields and outputting the plane grasping pose of each pixel. The trained pixel-level grasp pose detection network has the following network structure: The backbone network, denoising network, feature fusion network and decoder are connected in sequence; The backbone network is implemented using an improved Swin Transformer network; The improved Swin Transformer network is obtained by replacing all Swin Transformer Blocks of the Swin Transformer network with the first improved Swin Transformer Block; Among them, the first improved Swin Transformer Block includes: A first basic unit and a second basic unit connected in sequence; The first basic unit includes: a first normalization layer LN, a window multi-head self-attention mechanism layer, an adder J1, a second normalization layer LN, a first improved FFN module and an adder J2 connected in sequence; wherein the input end of the adder J1 is residually connected to the input end of the first normalization layer LN; the input end of the adder J2 is residually connected to the input end of the second normalization layer LN; The second basic unit includes: a third normalization layer LN, a shifted window multi-head self-attention mechanism layer, an adder J3, a fourth normalization layer LN, a second improved FFN module and an adder J4 connected in sequence; wherein the input end of the adder J3 is residually connected to the input end of the third normalization layer LN; the input end of the adder J4 is residually connected to the input end of the fourth normalization layer LN; The input end of the first normalization layer LN serves as the input end of the first improved Swin Transformer Block; the output end of the adder J4 serves as the output end of the first improved Swin Transformer Block; the output end of the adder J2 is connected to the input end of the third normalization layer LN; The internal structure of the first improved FFN module is consistent with that of the second improved FFN; The first improved FFN module includes: The 1*1 convolution layer C1, the dimension transformation Reshape layer R1, the 3*3 depth-wise separable convolution Depth-wise layer C2, the dimension transformation Reshape layer R2 and the convolution layer C3 are connected in sequence.

7. An electronic device, comprising: a memory for non-transitory storage of computer-readable instructions; as well as a processor for executing said computer-readable instructions, When the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is executed.

8. A storage medium, characterized in that: The computer-readable instructions are non-transitory stored, wherein when the computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1 to 5 are executed.

Images