A method for processing pleurotus eryngii surface image for robot grabbing

Abstract

The application discloses a kind of for robot grabbing's pleurotus eryngii surface image processing method, method mainly is using neural network to the incoming RGB-D pleurotus eryngii picture is handled to obtain multi-scale target feature tensor, through attention mechanism module, multi-scale feature tensor is handled, then through residual up-sampling module, through neural network output result, pleurotus eryngii coordinate position and the angle of capture are calculated and output.The application is improved to the feature extraction of pleurotus eryngii in image by using SetFeat network without increasing the amount of parameters, realizes the further improvement of pleurotus eryngii coordinate position positioning precision and the angle of capture precision.

Description

Technical Field

[0001] This invention relates to the fields of deep learning, image processing, and robotics, specifically to a method for improving the machine's recognition accuracy and grasping efficiency of king oyster mushrooms in images. Background Technology

[0002] For humans, grasping allows them to quickly and accurately pick up unknown objects based on their experience. Therefore, industrial robots are constantly compared to humans in terms of accuracy and speed of grasping. With the deepening research into robot vision, industry needs a universal technology to simulate the rapid and robust grasping of any type of object by robots. The main challenge lies in how to accurately transfer the knowledge learned by visual robots to new targets.

[0003] Current grasping robots utilize traditional machine learning methods such as FAST and SIFT image processing techniques, as well as deep learning models for visual robot grasping and detection. However, most methods transfer knowledge gained from a set of basic categories to a new set of targets, lacking expert knowledge about the targets themselves. This makes them unable to accurately identify and quickly grasp new targets. Therefore, most methods can only handle training sets of specific targets, requiring correct placement. If the new target has an angular deviation, the results are often unsatisfactory, especially for small objects. Furthermore, there are issues such as unclear target features and inaccurate grasping localization. This is because most models lack the ability to extract the target's position and deep features within the image, and to predict the target's angular deviation.

[0004] Arman et al. proposed a backbone network called SetFeat in their paper "Matching feature sets for few-shot image classification. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022:9014-9024." This network incorporates an attention mechanism module. The main function of this module is to process the output features of the neural network through attention mechanisms from different modules. This method enables the extraction of image feature sets rather than simply single features, and it processes these feature sets sequentially. However, when fusing features using the multi-head attention mechanism, global average pooling is used. For images with a single target, global average pooling increases noise in the target feature region. Therefore, global average pooling is removed and replaced with max pooling, improving image classification performance without increasing the total number of network parameters. However, while this method can handle target feature information in image classification and object detection maps, it lacks non-linear expressive power. Therefore, this invention improves the SetFeat backbone network by adding a LeakyReLU activation function module, thereby improving the model's accuracy.

[0005] Sulabh et al., in their paper "Antipodal robotic grasping using generative residual convolutional neural network" (In 2020 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS) 2020:9626-9633), proposed an end-to-end method for target grasping detection. This method includes fitting the feature map output by the neural network to generate the bounding box for the grasping target, and evaluating the accuracy of the grasping target parameters and the loss function. However, the neural network used in their model inference process still suffers from uncertainty and instability in the recognition and grasping of specific targets, as well as problems with ineffective feature extraction of the target's location region in RGB-D images and excessive noise. Therefore, they used SetFeat as the backbone network and employed its proposed target coordinate localization and grasping angle calculation methods to achieve accurate results for the target grasping coordinates and angle. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides a method for processing surface images of king oyster mushrooms for robot grasping. The model is mainly applied to improve the accuracy of the coordinate position and grasping angle of king oyster mushrooms in RGB-D images captured by industrial cameras.

[0007] The technical solution adopted in this invention is: a method for processing surface images of king oyster mushrooms for robot grasping, comprising the following steps:

[0008] S1: The input king oyster mushroom image is processed through the backbone network SetFeat to obtain the feature tensors of angle and width.

[0009] S2: The deviation angle of the king oyster mushroom relative to the horizontal angle is obtained by calculating the arctangent function of the angular feature tensor. Then, the relative coordinate position of the king oyster mushroom relative to the camera and the deviation angle are output by compressing the dimensions.

[0010] Based on the above scheme, as a preferred embodiment, the specific method of step S1 includes the following steps:

[0011] S101: Scale the input king oyster mushroom image to a size of 224×224;

[0012] S102: After passing the scaled king oyster mushroom image through four 3×3 convolution operations, normalization operations, and activation function operations, four feature tensors B1, B2, B3, and B4 are output.

[0013] S103: The obtained feature tensors B1, B2, B3, and B4 are mapped from their original scale to multi-scale feature tensors B1_1, B2_1, B2_2, B3_1, B3_2, B3_3, B4_1, B4_2, and B4_3. These are then input into the attention mechanism module, where two matrix multiplication operations, one normalization process, one feature concatenation process, and one max pooling process are performed. B1_1 yields the locally enhanced feature tensor A1; B2_1 and B2_2 yield the locally enhanced feature tensor A2; B3_1, B3_2, and B3_3 yield the locally enhanced feature tensor A3; and B4_1, B4_2, and B4_3 yield the locally enhanced feature tensor A4. Finally, the locally enhanced features A1, A2, A3, and A4 are fused with the feature tensor B4 to form ABTotal.

[0014] S104: Input the obtained fused feature ABTotal into the residual upsampling module to obtain the feature tensor AB_R_Ttoal;

[0015] S105: Take the feature tensor AB_R_Ttoal as input and perform a convolution operation to obtain the feature tensors of angle and width.

[0016] Based on the above scheme, as a preferred option, in step S102, four feature tensors are output: B1 with 64×64 channels, B2 with 128×128 channels, B3 with 256×256 channels, and B4 with 512×512 channels.

[0017] Based on the above scheme, as a preferred embodiment, in step S103, the four obtained feature tensors B1 (64×64 channels), B2 (128×128 channels), B3 (256×256 channels), and B4 (512×512 channels) are mapped from their original scale to multi-scale feature tensors B1_1, B2_1, B2_2, B3_1, B3_2, B3_3, B4_1, B4_2, and B4_3. These are then input into the attention mechanism module, where each feature tensor undergoes two matrix multiplication operations. After one normalization process, one feature concatenation process, and one max pooling process, the local enhancement feature tensor A1 is obtained from B1_1; the local enhancement feature tensor A2 is obtained from B2_1 and B2_2; the local enhancement feature tensor A3 is obtained from B3_1, B3_2, and B3_3; and the local enhancement feature tensor A4 is obtained from B4_1, B4_2, and B4_3. Then, the local enhancement features A1, A2, A3, and A4 are fused with the feature tensor B4 to form a feature tensor ABTotal with 64×64 channels and an image size of 14×14.

[0018] Based on the above scheme, as a preferred option, the attention mechanism module maps the multi-scale feature tensors B1_1, B2_1, B2_2, B3_1, B3_2, B3_3, B4_1, B4_2, and B4_3 to their original scales respectively. Obtained by mapping The original scale is then mapped to a three-layer feature tensor of q, k, and v. Matrix calculations and normalization operations are then performed on q and k to obtain the attention score. A matrix multiplication operation is performed with the feature tensor v mapped from the third layer, and then addition operations are performed with the input feature tensors B1_1, B2_1, B2_2, B3_1, B3_2, B3_3, B4_1, B4_2, and B4_3 respectively. In the attention feature fitting module, the normalization layer and global average pooling layer are removed, and a max pooling layer is added to improve the accuracy of the obtained feature map and reduce redundant noise in the feature map. The formula for calculating the attention score of the attention mechanism module is as follows:

[0019]

[0020] In the formula, express The feature mapper of the input attention mechanism feature map Indicates the scaling factor. express Attention score.

[0021] Based on the above scheme, as a preferred option, in In matrix multiplication with the third-level mapping v, matrix multiplication can be expressed using the following formula:

[0022]

[0023] In the formula, This represents the calculated dot product attention block. The result of the matrix product of the third layer v obtained by the mapping.

[0024] Based on the above scheme, as a preferred embodiment, the residual upsampling module uses four 2D convolutional neural networks with 3×3 kernels, three 2D convolutional neural networks with 1×1 kernels, four normalization functions, four LeakyReLU activation functions, and one residual structure. The first four sampling layers consist of four convolutions with 3×3 kernels, four normalization functions, and four LeakyReLU activation functions, respectively. The fifth sampling layer uses three convolutions with 1×1 kernels. The fused feature tensor ABTotal, with 64×64 channels and an image size of 14×14, is input into the residual upsampling module to perform the upsampling of the 64×64 channels and 14×14 image size. The small fusion feature tensor ABTotal is mapped to its original scale. On the one hand, the fusion feature tensor ABTotal is subjected to dimensionality increase operation through a convolution with a kernel size of 3×3, a normalization function, and a LeakyReLU activation function to obtain AB_total_1 with a channel size of 512×512, AB_total_2 with a channel size of 256×256, AB_total_3 with a channel size of 128×128, and AB_total_4 with an image size of 224×224 and a channel size of 64×64. Then, AB_total_4 is added to ABTotal to obtain AB_R_Ttoal with an image size of 224×224.

[0025] Based on the above scheme, as a preferred option, the relative coordinate position of the king oyster mushroom with respect to the camera and the deviation angle of the king oyster mushroom relative to the horizontal angle can be expressed by the following equation:

[0026]

[0027] In the formula, Indicates the center coordinates of the king oyster mushroom; This represents the measured angular rotation required for the king oyster mushroom at each point, ranging from [0, π]. Represents the required width, and represents a measure of uniform depth, in [0, ]express, Q represents the maximum width of the robot gripper; Q represents the gripping quality score.

[0028] Based on the above scheme, as a preferred option, the bounding boxes generated by the neural network for the king oyster mushroom are compared with the actual target bounding boxes for accuracy and loss calculation. The loss calculation formula used for the bounding boxes is as follows:

[0029]

[0030] in, This indicates the bounding box of the king oyster mushroom generated by the model; Represents the border of a real king oyster mushroom. Represented as:

[0031]

[0032] Beneficial effects: Compared with the prior art, the present invention has the following advantages: 1. The use of an attention mechanism module improves the accuracy of the coordinates, grasping angle, and feature regions of the king oyster mushroom in the image; 2. The feature extraction module added to the attention mechanism module and the backbone network SetFeat improves the neural network's ability to extract features of the king oyster mushroom in the image; 3. The use of a residual upsampling module enables the detection and grasping of king oyster mushrooms of various shapes in the image, improving the feature extraction capability and increasing the robustness and robustness of the model without increasing the computational load or parameters. Attached Figure Description

[0033] Figure 1 This is a framework diagram of the SetFeat neural network downsampling stage of the present invention;

[0034] Figure 2 This is a framework diagram of the lightweight attention mechanism module of the present invention;

[0035] Figure 3 This is a framework diagram of the residual upsampling stage of the present invention;

[0036] Figure 4 This is a flowchart of the present invention. Detailed Implementation

[0037] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0038] like Figure 1-4 As shown, S1: The input king oyster mushroom image is processed through the backbone network SetFeat to obtain the feature tensors of angle and width.

[0039] Furthermore, the aforementioned method for processing images of the surface of king oyster mushrooms for robot grasping comprises the following steps:

[0040] S101: Scale the input king oyster mushroom image to a size of 224×224;

[0041] S102: The scaled king oyster mushroom image is sequentially passed through the backbone network SetFeat, followed by four 3×3 convolution operations, a normalization operation, and an activation function operation, resulting in four feature tensors: B1 with 64×64 channels, B2 with 128×128 channels, B3 with 256×256 channels, and B4 with 512×512 channels.

[0042] Furthermore, such as Figure 1 As shown, the backbone network SetFeat maximizes the extraction of salient features of the target in the image. The target feature map obtained by processing each part through the convolutional neural network module is passed as input to the attention mechanism module corresponding to each module for processing.

[0043] S103: The four obtained feature tensors B1 (64×64 channels), B2 (128×128 channels), B3 (256×256 channels), and B4 (512×512 channels) are mapped from their original scale to multi-scale feature tensors B1_1, B2_1, B2_2, B3_1, B3_2, B3_3, B4_1, B4_2, and B4_3. These are then input into the attention mechanism module, where each feature tensor undergoes two matrix multiplications and one normalization operation. After one feature concatenation and one max pooling process, B1_1 yields the local enhancement feature tensor A1; B2_1 and B2_2 yield the local enhancement feature tensor A2; B3_1, B3_2, and B3_3 yield the local enhancement feature tensor A3; and B4_1, B4_2, and B4_3 yield the local enhancement feature tensor A4. Then, the local enhancement features A1, A2, A3, and A4 are fused with the feature tensor B4 to form a feature tensor ABTotal with 64×64 channels and an image size of 14×14.

[0044] Furthermore, the formula for calculating the attention score of the attention mechanism module is as follows:

[0045]

[0046] Furthermore, attention mechanism modules such as Figure 2 As shown, the multi-scale feature tensors B1_1, B2_1, B2_2, B3_1, B3_2, B3_3, B4_1, B4_2, and B4_3 are mapped to their original scales respectively. Obtained by mapping The original scale is then mapped to a three-layer feature tensor of q, k, and v. Matrix calculations and normalization operations are then performed on q and k to obtain the attention score. Matrix multiplication is performed with the feature tensor v mapped in the third layer, and addition is performed with the input feature tensors B1_1, B2_1, B2_2, B3_1, B3_2, B3_3, B4_1, B4_2, and B4_3 respectively. In the fitting attention feature module, the normalization layer and the global average pooling layer are removed, and the max pooling layer is added to improve the accuracy of the obtained feature map and reduce redundant noise in the feature map.

[0047] In the formula, express The feature mapper of the input attention mechanism feature map Indicates the scaling factor. express Attention score.

[0048] Furthermore, in In matrix multiplication with the third-level mapping v, matrix multiplication can be expressed using the following formula:

[0049]

[0050] In the formula, This represents the calculated dot product attention block. The result of the matrix product of the third layer v obtained by the mapping.

[0051] Furthermore, the formula for calculating the attention score of the attention mechanism module is as follows:

[0052]

[0053] In the formula, express The feature mapper of the input attention mechanism feature map Indicates the scaling factor. express Attention score.

[0054] Furthermore, in In matrix multiplication with the third-level mapping v, matrix multiplication can be expressed using the following formula:

[0055]

[0056] In the formula, This represents the calculated dot product attention block. The result of the matrix product of the third layer v obtained by the mapping.

[0057] S104: Input the fused feature tensor ABTotal, which has 64×64 channels and an image size of 14×14, into the residual upsampling module to obtain the feature tensor AB_R_Ttoal;

[0058] Furthermore, such as Figure 3 The diagram shows a residual upsampling module designed according to this method. Through the residual structure, the residual upsampling module consists of four 2D convolutional neural networks with 3×3 kernels, three 2D convolutional neural networks with 1×1 kernels, four normalization functions, four LeakyReLU activation functions, and one residual structure. The first four sampling layers consist of four 3×3 convolutions, four normalization functions, and four LeakyReLU activation functions, respectively. The fifth sampling layer uses three 1×1 convolutions. The fused feature tensor ABTotal, with 64×64 channels and an image size of 14×14, is input into the residual upsampling module. On one hand, the fused feature tensor ABTotal is mapped to its original scale. On the other hand, the fused feature tensor ABTotal is upsized by a convolution with a kernel size of 3×3, a normalization function, and a LeakyReLU activation function, resulting in AB_total_1 with a channel size of 512×512, AB_total_2 with a channel size of 256×256, AB_total_3 with a channel size of 128×128, and AB_total_4 with an image size of 224×224 and a channel size of 64×64. Then, AB_total_4 is added to ABTotal to obtain AB_R_Ttoal with an image size of 224×224. This effectively improves the accuracy of identifying the feature region where the king oyster mushroom is located and the target features in the image; the residual upsampling module improves the saliency of the feature map of the target region and the accuracy of target localization through residual convolution operation; the residual upsampling module processes the fitted multi-scale features to output the coordinates of the king oyster mushroom, the grasping angle and the feature region in the image; the feature information obtained by upsampling is processed by Gaussian filtering to eliminate redundant noise around the king oyster mushroom.

[0059] S105: Take the feature tensor AB_R_Ttoal as input and perform a convolution operation to obtain the feature tensors of angle and width.

[0060] S2: By calculating the arctangent function value of the angular feature tensor, the relative coordinate position of the king oyster mushroom with respect to the camera and the deviation angle of the king oyster mushroom relative to the horizontal angle are output after compressing the dimensions. The relative coordinate position of the king oyster mushroom with respect to the camera and the deviation angle of the king oyster mushroom relative to the horizontal angle can be expressed by the following equation:

[0061]

[0062] In the formula, Indicates the center coordinates of the king oyster mushroom; This represents the measured angular rotation required for the king oyster mushroom at each point, ranging from [0, π]. Represents the required width, and represents a measure of uniform depth, in [0, ]express, represents the maximum width of the robot gripper; Q represents the feature tensor AB_R_Ttoal obtained by passing the image through the model and mapping it onto the image, that is, the value of each pixel of the king oyster mushroom on the image, with the value between 0 and 1, and the closer it is to 1, the greater the probability of grasping.

[0063] Furthermore, in this method, the accuracy and loss are calculated for the bounding boxes generated by the neural network for the king oyster mushroom and the actual target bounding boxes. The loss calculation formula used for the bounding boxes is shown below:

[0064]

[0065] in, This indicates the bounding box of the king oyster mushroom generated by the model; Represents the border of a real king oyster mushroom. Represented as: ,

[0066] Furthermore, the optimizer used is the Adam optimizer, which performs backward optimization layer by layer based on the gradient of each convolutional module.

[0067] The robustness, feasibility, and effectiveness of the method described were tested using images of king oyster mushrooms. The method of this invention achieved an accuracy of 83.07% when comparing the relative coordinate position of the king oyster mushroom with respect to the camera and the deviation angle of the king oyster mushroom from the horizontal angle with a manually calibrated relative coordinate position of the mushroom with respect to the camera, which is 3% higher than the accuracy achieved using ordinary residual module convolution.

Claims

1. A method for image processing of the surface of king oyster mushrooms for robot grasping, characterized in that, It consists of the following steps: S1: The input king oyster mushroom image is processed through the backbone network SetFeat to obtain the feature tensors of angle and width. S101: Scale the input king oyster mushroom image to a size of 224×224; S102: After the scaled king oyster mushroom image is processed by four 3×3 convolution operations, normalization operations, and activation function operations, four feature tensors B1, B2, B3, and B4 are output. S103: The obtained feature tensors B1, B2, B3, and B4 are mapped from their original scale to multi-scale feature tensors B1_1, B2_1, B2_2, B3_1, B3_2, B3_3, B4_1, B4_2, and B4_3. These are then input into the attention mechanism module, where two matrix multiplications, one normalization operation, one feature concatenation operation, and one max pooling operation are performed. B1_1 yields the locally enhanced feature tensor A1; B2_1 and B2_2 yield the locally enhanced feature tensor A2; B3_1, B3_2, and B3_3 yield the locally enhanced feature tensor A3; and B4_1, B4_2, and B4_3 yield the locally enhanced feature tensor A4. Finally, the locally enhanced features A1, A2, A3, and A4 are fused with feature tensor B4 to form ABTotal. Specifically, the attention mechanism module maps the multi-scale feature tensors B1_1, B2_1, B2_2, B3_1, B3_2, B3_3, B4_1, B4_2, and B4_3 to their original scales. Obtained by mapping The original scale is then mapped to a three-layer feature tensor of q, k, and v. Matrix calculations and normalization operations are then performed on q and k to obtain the attention score. Matrix multiplication is performed with the feature tensor v mapped in the third layer, and addition is performed with the input feature tensors B1_1, B2_1, B2_2, B3_1, B3_2, B3_3, B4_1, B4_2, and B4_3 respectively. The normalization layer and the global average pooling layer are removed in the fitting attention feature module. S104: Input the obtained fused feature ABTotal into the residual upsampling module to obtain the feature tensor AB_R_Ttoal; S105: Take the feature tensor AB_R_Ttoal as input and perform a convolution operation to obtain the feature tensors of angle and width. S2: The deviation angle of the king oyster mushroom relative to the horizontal angle is obtained by calculating the arctangent function value of the angular feature tensor. Then, the relative coordinate position of the king oyster mushroom relative to the shooting camera and the deviation angle are output by compressing the dimensions.

2. The method for processing the surface image of king oyster mushrooms for robot grasping according to claim 1, characterized in that, In step S102, four feature tensors are output: B1 with 64×64 channels, B2 with 128×128 channels, B3 with 256×256 channels, and B4 with 512×512 channels.

3. The method for processing images of king oyster mushroom surfaces for robot grasping according to claim 2, characterized in that, In step S103, the four obtained feature tensors B1 (64×64 channels), B2 (128×128 channels), B3 (256×256 channels), and B4 (512×512 channels) are mapped from their original scale to multi-scale feature tensors B1_1, B2_1, B2_2, B3_1, B3_2, B3_3, B4_1, B4_2, and B4_3. These are then input into the attention mechanism module, where each feature tensor undergoes two matrix multiplication operations and one normalization operation. After one feature concatenation and one max pooling operation, B1_1 yields the local enhancement feature tensor A1; B2_1 and B2_2 yield the local enhancement feature tensor A2; B3_1, B3_2, and B3_3 yield the local enhancement feature tensor A3; and B4_1, B4_2, and B4_3 yield the local enhancement feature tensor A4. Then, the local enhancement features A1, A2, A3, and A4 are fused with the feature tensor B4 to form a feature tensor ABTotal with 64×64 channels and an image size of 14×14.

4. The method for processing images of king oyster mushroom surfaces for robot grasping according to claim 1, characterized in that, The formula for calculating the attention score of the attention mechanism module is as follows: ， In the formula, express The feature mapper of the input attention mechanism feature map Indicates the scaling factor. express Attention score.

5. The method for processing the surface image of king oyster mushrooms for robot grasping according to claim 4, characterized in that, exist In matrix multiplication with the third-level mapping v, matrix multiplication can be expressed using the following formula: ， In the formula, This represents the calculated dot product attention block. The result of the matrix product of the third layer v obtained by the mapping.

6. The method for processing the surface image of king oyster mushrooms for robot grasping according to claim 5, characterized in that, The residual upsampling module consists of four 2D convolutional neural networks with 3×3 kernels, three 2D convolutional neural networks with 1×1 kernels, four normalization functions, four LeakyReLU activation functions, and one residual structure. The first four sampling layers consist of four 3×3 convolutions, four normalization functions, and four LeakyReLU activation functions, respectively. The fifth sampling layer uses three 1×1 convolutions. When the fused feature tensor ABTotal with 64×64 channels and 14×14 image size is input into the residual upsampling module, it performs the fusion feature tensor... The ABTotal is mapped to its original scale. First, the fused feature tensor ABTotal is subjected to dimensionality increase operation through a 3×3 convolution kernel, a normalization function, and a LeakyReLU activation function to obtain AB_total_1 with a channel size of 512×512, AB_total_2 with a channel size of 256×256, AB_total_3 with a channel size of 128×128, and AB_total_4 with an image size of 224×224 and a channel size of 64×64. Then, AB_total_4 is added to ABTotal to obtain AB_R_Ttoal with an image size of 224×224.

7. The method for processing the surface image of king oyster mushrooms for robot grasping according to claim 6, characterized in that, The relative coordinates of the king oyster mushroom with respect to the camera and its deviation from the horizontal angle can be expressed by the following equations: ， In the formula, Indicates the center coordinates of the king oyster mushroom; This represents the measured angular rotation required for the king oyster mushroom at each point, ranging from [0,π]. Represents the required width, and represents a measure of uniform depth, in [0, ]express, Q represents the maximum width of the robot gripper; Q represents the gripping quality score.

8. The method for processing the surface image of king oyster mushrooms for robot grasping according to claim 6, characterized in that, The bounding boxes generated by the neural network for the king oyster mushroom are compared with the bounding boxes of the real target to calculate accuracy and loss. The loss calculation formula used for the bounding boxes is shown below: ， in, This indicates the bounding box of the king oyster mushroom generated by the model; This represents the border of a real king oyster mushroom. Represented as:

Images