A regional attention semantic segmentation method and system based on encoding and decoding structure

Abstract

The present invention provides a method and system for regional attention semantic segmentation based on a codec structure, comprising the following steps: inputting an acquired image to be segmented into a pre-constructed regional attention network model based on a codec structure to obtain a segmentation map; wherein the pre-constructed regional attention network model structure based on the codec structure includes a feature extraction network, a regional attention module and a multi-scale jump feature fusion module; the method of the present invention not only effectively solves the problem of mis-segmentation of occluded objects, but also can improve the accurate expression of edge features of objects to be segmented, thereby improving segmentation accuracy and achieving accurate segmentation of objects.

Description

Technical Field

[0001] The present invention belongs to the technical field of object detection, and specifically relates to a regional attention semantic segmentation method and system based on a codec structure. Background Art

[0002] Object detection technology is an important component of robot vision systems. Its main task is to process captured images, extract the object's location and category information, and control the robot to grasp it based on this information. Therefore, object detection technology is particularly important in robot vision systems.

[0003] Industrial robot assembly operations require extremely high grasping accuracy; deviations in grasping can lead to assembly failure. Traditional object grasping methods use image processing to generate a segmentation map of the part and its background. Edge detection is then applied to the segmentation map to determine the part's minimum bounding rectangle. The grasping points are then calculated from the four vertices of the rectangle.

[0004] Traditional image processing methods are easily affected by environmental factors and cannot guarantee the complete segmentation of parts and background, which directly leads to deviations in the grasping points calculated based on the segmentation map, resulting in grasping errors. Summary of the Invention

[0005] The purpose of the present invention is to provide a regional attention semantic segmentation method and system based on a codec structure, which overcomes the defect of large errors in existing image processing methods.

[0006] In order to achieve the above object, the technical solution adopted in the present invention is:

[0007] The present invention provides a method for regional attention semantic segmentation based on a codec structure, comprising the following steps:

[0008] The acquired image to be segmented is input into a pre-built regional attention network model based on a codec structure to obtain a segmentation map; wherein the pre-built regional attention network model structure based on a codec structure includes a feature extraction network, a regional attention module and a multi-scale jump feature fusion module.

[0009] Preferably, the feature extraction network is a Resnet50 network.

[0010] Preferably, the regional attention module includes multiple convolutional layers, upsampling layers and loss functions, and each convolutional layer is followed by a batch normalization layer BN and a nonlinear activation function ReLu.

[0011] Preferably, the upsampling layer uses bilinear interpolation; and the loss function adopts intersection-over-union loss.

[0012] Preferably, the multi-scale jump feature fusion module includes a 1×1 convolution layer and upsampling, and the 1×1 convolution layer consists of convolution, a batch normalization layer BN and a nonlinear activation function ReLu.

[0013] Preferably, the upsampling layer uses bilinear interpolation.

[0014] Preferably, the output of the multi-scale jump feature fusion module is connected to the convolution layer, the output of the convolution layer is connected to the upsampling layer, and the output of the upsampling layer is the segmentation result of the regional attention network based on the codec structure.

[0015] Preferably, the total loss function of the pre-built regional attention network model based on the codec structure is:

[0016] Loss = IoU loss + Cross loss

[0017] Among them, IoU loss is intersection-over-union loss, and Corss loss is cross entropy loss.

[0018] A regional attention semantic segmentation system based on an encoder-decoder structure, comprising:

[0019] A data preprocessing module is used to preprocess the acquired data set to obtain a preprocessed data set;

[0020] Network module, used to pre-build the regional attention network model based on the encoder-decoder structure;

[0021] The segmentation module is used to input the acquired image to be segmented into a pre-built regional attention network model based on the codec structure to obtain a segmentation map.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] The present invention provides a regional attention semantic segmentation method based on a codec structure, which uses the Resnet50 feature extraction network to extract key information in the image. First, the global branch uses the regional attention module to roughly segment the approximate area where the object is located, effectively solving the problem of mis-segmentation of occluded objects; secondly, the local branch adopts a multi-scale jump feature fusion strategy to perform fine segmentation on the image and enhance the edge features of the objects to be segmented; finally, the global and local branches are fused to obtain the final feature map, and the feature map can be upsampled to obtain the segmentation map.

[0024] In summary, the method of the present invention not only effectively solves the problem of mis-segmentation of occluded objects, but also improves the accurate expression of edge features of the objects to be segmented, thereby improving the segmentation accuracy and achieving accurate segmentation of objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Flowchart of the regional attention network based on the encoder-decoder structure;

[0026] Figure 2 This is a diagram of the regional attention network structure based on the encoder-decoder structure;

[0027] Figure 3 This is the structural diagram of the regional attention module;

[0028] Figure 4 This is the structure diagram of the multi-scale jump feature fusion module;

[0029] Figure 5 The segmentation results of different semantic segmentation methods. DETAILED DESCRIPTION

[0030] In recent years, semantic segmentation methods have been widely used in remote sensing imaging, medical imaging, unmanned driving and other fields. Therefore, semantic segmentation methods are applied to the research of robot grasping and recognition in order to better separate objects from the background in the image.

[0031] The present invention is described in further detail below with reference to the accompanying drawings:

[0032] See also Figure 1 , the present invention proposes a regional attention semantic segmentation method based on the codec structure, comprising the following steps:

[0033] Step 1. Build a remote control parts dataset. The main steps are:

[0034] Collect images of remote control parts and build a remote control parts dataset. The dataset includes remote control parts in different postures, different backgrounds, different lighting, and stacking.

[0035] In the specific implementation of the present invention, the remote control parts data set is divided into a training set and a test set in a ratio of 7:3. The training set is used to train the parameters in the network model, and the test set is used to evaluate the performance of the method.

[0036] Step 2. Construct a regional attention network based on the encoder-decoder structure. The main steps are:

[0037] Figure 2 This is the structural diagram of the regional attention network based on the codec structure. The regional attention network based on the codec structure consists of a feature extraction network, a regional attention module, and a multi-scale jump feature fusion module. The feature extraction network adopts the Resnet50 network.

[0038] After preprocessing the remote control parts dataset in step 1, it is input into the feature extraction network to output a feature map F1 of size 2048×h×w. First, the feature map F1 is processed using the regional attention module to obtain a feature map F5, which focuses more on the area where the object is located; then the multi-scale jump feature fusion module is used to fuse the feature map F5 with the shallow features of the feature extraction network to obtain the feature map F 11 , the feature map not only contains the area information of the object, but also enhances the edge features of the object; finally, convolution is used to transform the feature map F 11 Fine-tune the network and restore the final feature map to the original size by upsampling to obtain the network output result.

[0039] Step 3. The regional attention module guides the network to focus on the area where the object is located. Its main steps are:

[0040] Figure 3 This is the structural diagram of the regional attention module. The regional attention module includes multiple convolutional layers, upsampling layers and loss functions. Each convolutional layer is followed by a batch normalization layer BN and a nonlinear activation function ReLu. The upsampling layer uses bilinear interpolation, and the loss function uses intersection-over-union loss.

[0041] First, the feature map F1 in step 2 is channel compressed using 1×1 convolution, and the size of the compressed feature map F2 is 256×h×w.

[0042] Next, we perform 1×1 convolution on feature map F2 to reduce its dimensionality. The resulting feature map F3 is of size C×h×w, where C represents the number of categories. Upsampling is used to restore feature map F3 to its original size, and the intersection-over-union loss is calculated to force the network to learn the corresponding region for each channel feature map. Feature maps F3 are then superimposed according to the channel dimension to obtain a 1×h×w feature map F4.

[0043] Finally, since the feature map after the dot product operation of feature map F2 and feature map F4 only contains the approximate area where the object is located and lacks the information of the object edge, the feature map after the dot product and feature map F2 need to be spliced ​​by channel to output the feature map F5. This feature map focuses on the information of the area where the object is located and retains the object edge information as much as possible, effectively solving the problem of mis-segmentation of occluded objects.

[0044] Step 4. Calculate the intersection-over-union loss for the feature map F3 in step 3. The calculation formula is:

[0045] For the feature map of each channel in the feature map F3, an intersection-over-union loss is calculated:

[0046]

[0047] Among them, H is the height of the picture, W is the width of the picture, p ij Indicates the label value of row i and column j, y ij Represents the predicted value of row i and column j. The intersection-over-union loss of each class is summed and averaged to obtain the final loss value.

[0048] Step 5. The multi-scale jump feature fusion module improves the accurate expression of the edge features of the object to be segmented. The main steps are:

[0049] See also Figure 4 This is a diagram of the multi-scale skip feature fusion module. The decoding part introduces multi-scale skip feature fusion, which uses the rich local information of shallow feature maps to improve the object edge segmentation effect.

[0050] The multi-scale jump feature fusion module includes a 1×1 convolutional layer and upsampling. The 1×1 convolutional layer consists of convolution, batch normalization (BN), and a nonlinear activation function (ReLu). The upsampling layer uses bilinear interpolation. 1×1 convolution is used to reduce the channel dimension of the shallow feature map output by the feature extraction network. A 2x upsampling is used to scale the deep feature map to the size of the shallow feature map.

[0051] First, the feature map F5 in step 3 is upsampled by a factor of 2, and the shallow feature map F6 output by the feature extraction network is subjected to channel dimensionality reduction using a 1×1 convolution. The two feature maps are concatenated according to the channel dimension to obtain the feature map F7.

[0052] Secondly, the feature map F7 is upsampled by a factor of 2, and the shallow feature map F8 output by the feature extraction network is subjected to channel dimensionality reduction through 1×1 convolution. The two feature maps are concatenated according to the channel dimension to obtain the feature map F9.

[0053] Finally, the feature map F9 is upsampled by 2 times, and the shallow feature map F output by the feature extraction network is processed by 1×1 convolution. 10 Perform channel dimensionality reduction and concatenate these two feature maps according to the channel dimension to obtain the feature map F 11 ;

[0054] Step 6. The feature map F output by the multi-scale jump feature fusion module 11 With rich object information. 11 3×3 convolution is used for fine-tuning, and then the feature map output after fine-tuning is restored to the original image size using a 4-fold upsampling operation to obtain the segmentation result of the network, and the cross entropy loss is calculated for the segmentation result.

[0055] Step 7. Calculate the cross entropy loss for the segmentation results in step 6. The calculation formula is:

[0056] The cross entropy loss for each channel in the segmentation result is:

[0057]

[0058] Among them, H is the height of the picture, W is the width of the picture, p ij represents the true value of row i and column j, q ij Represents the predicted value of the i-th row and j-th column. The cross entropy loss of each channel is summed up, and the average of the sum is the loss value of the feature map F8.

[0059] Step 8. The total loss function of the regional attention network based on the encoder-decoder structure is:

[0060] Loss = IoU loss + Cross loss

[0061] Among them, IoU loss is intersection-over-union loss, and Corss loss is cross entropy loss.

[0062] Step 9. Call the trained model for prediction, preprocess the image to be segmented, and then input it into the regional attention network based on the encoder-decoder structure to output the final segmentation map.

[0063] In another embodiment of the present invention, a regional attention semantic segmentation system based on a codec structure is provided, which can be used to implement the above-mentioned regional attention semantic segmentation method based on a codec structure. The system includes a data preprocessing module, a network module, and a loss function module.

[0064] The data preprocessing module is used to set the image size to the same size, divide the remote control parts dataset into a training set and a test set according to category information, and perform data augmentation on the training set;

[0065] The network module is used to design a regional attention network based on the codec structure. The network includes a backbone network, a regional attention network, and a multi-scale skip feature fusion network. The training set obtained by the data preprocessing module is input into the backbone network using batch training to obtain a deep feature map. The regional attention network is used on the deep feature map to obtain the approximate area where the object is located. The shallow feature map and the deep feature map in the backbone network are input into the multi-scale skip feature fusion network to enhance the edge information of the object.

[0066] The loss function module is used to evaluate the distance between the predicted results and the actual values ​​using the sum of the intersection-over-union loss and the cross-entropy loss.

[0067] Table 1 Results of the proposed method and the existing method on the remote control dataset

[0068] Table 1 Results of different semantic segmentation methods

[0069]

[0070] Table 1 compares the proposed method with mainstream methods on a remote control dataset. The results show that the proposed codec-based regional attention segmentation method outperforms other networks, achieving a pixel accuracy of 99.75% and a mean intersection over union (MIoU) of 97.54%. The prediction time for a single image is reduced by 3.3 milliseconds. Overall, the proposed method achieves impressive segmentation accuracy.

[0071] See also Figure 5 Figures 1 and 2 show the segmentation results of different semantic segmentation methods. The first row shows the original image from the remote control parts dataset, the second row shows the segmentation results using the UNet network, the third row shows the segmentation results using the FCN-8s network, the fourth row shows the segmentation results using the PSPNet network, the fifth row shows the segmentation results using the DeeplabV3+ network, the sixth row shows the segmentation results using the proposed method, and the seventh row shows the ground truth labels. The figures clearly show that the proposed regional attention network based on the encoder-decoder structure can effectively address the problem of mis-segmentation of occluded objects and achieves better segmentation results for edge information.

[0072] In summary, the present invention is a regional attention semantic segmentation method based on a codec structure, the core of which is that the global branch uses a regional attention module to roughly segment the approximate area where the object is located, and the local branch uses a multi-scale jump feature fusion strategy to refine the edge of the object, improve the edge features of the object to be segmented, and finally fuse the global and local features to achieve accurate segmentation of the object. The present invention effectively improves the accuracy of object segmentation and plays a positive role in the development of the image segmentation field. The above content is only to illustrate the technical ideas of the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical ideas proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A regional attention semantic segmentation method based on an encoder-decoder structure, characterized in that: The following steps are involved: The acquired image to be segmented is input into a pre-built regional attention network model based on a codec structure to obtain a segmentation map; wherein the pre-built regional attention network model based on a codec structure includes a feature extraction network, a regional attention module and a multi-scale jump feature fusion module, and the specific method is: A feature extraction network is used to extract image information from the image to be segmented. The global branch uses the regional attention module to roughly segment the approximate area where the object is located. The local branch adopts a multi-scale jump feature fusion strategy to perform fine segmentation on the image. Finally, the global and local branches are fused to obtain the final feature map, and the feature map is upsampled to obtain the segmentation map.

2. The regional attention semantic segmentation method based on the codec structure according to claim 1 is characterized in that The feature extraction network is a Resnet50 network.

3. The regional attention semantic segmentation method based on the codec structure according to claim 1 is characterized in that The regional attention module includes multiple convolutional layers, upsampling layers and loss functions, and each convolutional layer is followed by a batch normalization layer BN and a nonlinear activation function ReLu.

4. The regional attention semantic segmentation method based on the codec structure according to claim 3 is characterized in that The upsampling layer uses bilinear interpolation; the loss function adopts intersection-over-union loss.

5. The regional attention semantic segmentation method based on the codec structure according to claim 1 is characterized in that The multi-scale jump feature fusion module includes a 1×1 convolutional layer and an upsampling layer. The 1×1 convolutional layer consists of a convolution, a batch normalization layer BN and a nonlinear activation function ReLu.

6. The method for regional attention semantic segmentation based on codec structure according to claim 5, characterized in that: The upsampling layer uses bilinear interpolation.

7. The regional attention semantic segmentation method based on the codec structure according to claim 1 is characterized in that The output of the multi-scale jump feature fusion module is connected to the convolution layer, the output of the convolution layer is connected to the upsampling layer, and the output of the upsampling layer is the segmentation result of the regional attention network based on the codec structure.

8. The regional attention semantic segmentation method based on the codec structure according to claim 1 is characterized in that The total loss function of the pre-built regional attention network model based on the encoder-decoder structure is: Among them, IoU loss is intersection-over-union loss, and Corss loss is cross entropy loss.

9. A regional attention semantic segmentation system based on an encoder-decoder structure, characterized in that: Based on the segmentation method according to claim 1, the system comprises: A data preprocessing module is used to preprocess the acquired data set to obtain a preprocessed data set; Network module, used to pre-build the regional attention network model based on the encoder-decoder structure; The segmentation module is used to input the acquired image to be segmented into a pre-built regional attention network model based on the codec structure to obtain a segmentation map.

Images