Shielding perception segmentation method based on RCF-Pix2Pix deep network

Abstract

The invention discloses an occlusion perception segmentation method based on an RCF-Pix2Pix deep network, and belongs to the technical field of digital image processing. The method comprises the following steps: acquiring an edge contour of a target in an original image X by using an RCF network to obtain an edge information graph P; respectively training a generator and a discriminator by using the improved loss function; and generating a target segmentation image by using the trained generator. According to the method, the RCF edge extraction network is combined with Pix2Pix, and the discriminator performs authenticity judgment by using different condition information, so that the attention in the edge region is enhanced; meanwhile, the edge information and the overall contour are combined as guidance, so that the Pix2Pix can complete the segmentation task relatively accurately even if the target contour is interfered; experimental results prove that compared with an existing segmentation method, the method shows higher segmentation precision in the aspect of shielding boundary segmentation.

Description

Technical Field

[0001] The invention relates to an occlusion-aware segmentation method based on an RCF-Pix2Pix deep network, and belongs to the technical field of digital image processing. Background Art

[0002] In recent years, visual recognition tasks such as image classification, object detection, edge detection, and semantic segmentation have achieved relatively successful results, which benefited from the development of large-scale data sets and deep learning technology. The annotation process of most data sets does not take into account the need to indicate occluded areas, so they cannot be used for non-modal perception and cannot well predict the shape contours of occluded areas. However, in some fields, being able to predict the complete contours of objects is crucial for understanding the scene. For example, in the field of agricultural production, by successfully predicting the shape of fruits obscured by leaves under different lighting conditions, the stability and accuracy of the robot in performing fruit picking tasks can be improved. Similarly, in autonomous driving technology, enabling cars to have the ability to infer the complete shape of obscured vehicles and pedestrians within the line of sight can significantly reduce the risk of collision. In addition, in industrial production scenarios, since products may overlap, have debris residues, or deform their shapes due to defects, being able to predict the original contours of products is crucial for extracting target surface information and ensuring accurate and stable grasping by the robot.

[0003] At present, there are some improvements on convolutional neural networks to achieve occluded object segmentation, but these solutions generally have problems such as complex and time-consuming training, poor robustness of multi-scale object segmentation, and large data set requirements. In addition to convolutional neural networks, generative adversarial networks (GANs) are also used in image segmentation tasks. Mazumdar, H. et al. used the GAN-based image conversion framework Pix2Pix to develop an efficient and reliable polyp segmentation method (Mazumdar, H., Chakraborty, C., Sathvik, M., Jayakumar, P., Kaushik, A.: Optimizing pix2pix gan with attention mechanisms for ai-driven polyp segmentation in iomt-enabled smarthealthcare, 2023); Chen, Z. et al. compared the trunk segmentation capabilities of Pix2Pix, U-Net and DeepLabv3 in images occluded by leaves, and the results showed that Pix2Pix achieved the highest segmentation accuracy (Chen, Z., Ting, D., Newbury, R., Chen, C.: Semantic segmentation for partially occluded apple trees based on deep learning, 2021). However, Pix2Pix has certain instability in segmenting edge areas, especially when the contour is missing in a large area, which will lead to obvious differences between the edge of the segmentation result and the actual edge. Summary of the invention

[0004] In order to enhance the segmentation accuracy of the occluded area, the present invention provides an occlusion-aware segmentation method based on the RCF-Pix2Pix deep network, and the technical solution is as follows:

[0005] The occlusion-aware target segmentation method of the present invention comprises:

[0006] Step 1: Use the RCF network to obtain the edge contour of the target in the original image X and obtain the edge information graph P;

[0007] Step 2: Train the generator;

[0008] Keep the parameters of the discriminator fixed, and transform the edge information graph P and the generated image G(X att ) is input into the discriminator for training, where X att =λX+(1-λ)P represents the edge feature enhancement map, λ represents the fusion weight;

[0009] Step 3: Train the discriminator;

[0010] Keep the parameters of the generator fixed, fuse the original image X and the edge information map P, and then merge the fused label image Y with the generated image G(X att ) input the discriminator for training;

[0011] Step 4: Use the trained generator to generate the target segmentation image.

[0012] Optionally, the loss function of the training generator in step 2 is:

[0013]

[0014] in, represents the least squares conditional GAN ​​loss, represents the MS-SSIM-L1 loss, Represents marginal loss.

[0015] Optionally, the loss function of the training discriminator in step 3 is:

[0016]

[0017] Where n represents the number of blocks after the image is divided, N, T 0 represents an N*N matrix filled with 0, T norm represents an N*N matrix filled with 1, D(Y) represents the result image after the real image passes through the discriminator, D(G(X att )) represents the result image after the fake image produced by the generator passes through the discriminator.

[0018] Optionally, the fusion weight λ is 0.9.

[0019] The present invention provides an occlusion-aware target segmentation system, comprising:

[0020] The edge extraction module is configured to obtain the edge contour of the target in the original image X by using the RCF network to obtain the edge information map P;

[0021] The image generation module is configured to keep the parameters of the discriminator fixed and transform the edge information map P and the generated image G(X att ) input to train the discriminator, where X att =λX+(1-λ)P represents the edge feature enhancement map, λ represents the fusion weight;

[0022] The image identification module is configured to keep the parameters of the generator fixed, fuse the original image X and the edge information map P, and then fuse the fused label image Y with the generated image G(Xatt ) input to train the discriminator;

[0023] The object segmentation module is configured to generate an object segmentation image using the trained generator.

[0024] Optionally, the loss function of the image generation module training generator is:

[0025]

[0026] in, represents the least squares conditional GAN ​​loss, represents the MS-SSIM-L1 loss, Represents marginal loss.

[0027] Optionally, the loss function of the image identification module training discriminator is:

[0028]

[0029] Where n represents the number of blocks after the image is divided, N, T 0 represents an N*N matrix filled with 0, T norm represents an N*N matrix filled with 1, D(Y) represents the result image after the real image passes through the discriminator, D(G(X att )) represents the result image after the fake image produced by the generator passes through the discriminator.

[0030] Optionally, the fusion weight λ is 0.9.

[0031] The present invention provides an occlusion-aware target segmentation device, comprising a memory and a processor;

[0032] The memory is used to store computer programs;

[0033] The processor is used to implement the occlusion-aware target segmentation method as described in any one of the above items when executing the computer program.

[0034] The present invention provides a computer-readable storage medium, characterized in that a computer program is stored on the storage medium, and when the computer program is executed by a processor, the occlusion-aware target segmentation method as described in any one of the above items is implemented.

[0035] The beneficial effects of the present invention are:

[0036] The present invention designs an edge-enhanced Pix2Pix generative adversarial network RCF-Pix2Pix, which combines the RCF edge extraction network with Pix2Pix and uses different conditional information to allow the discriminator to make authenticity judgments, so as to enhance the focus on edge areas. At the same time, the edge information and the overall contour are combined as guidance, so that Pix2Pix can complete the segmentation task relatively accurately even when the target contour is disturbed. Experimental results show that compared with the existing segmentation methods, the present invention shows higher segmentation accuracy in segmenting occluded boundaries. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 This is the RCF-Pix2Pix network structure diagram of the present invention.

[0039] Figure 2 It is a training loss curve diagram after the network is gradually optimized in the second embodiment of the present invention.

[0040] Figure 3 This is a visualization diagram of the effects of the present invention and other existing image segmentation schemes. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0042] Embodiment 1:

[0043] This embodiment provides an occlusion-aware target segmentation method, including:

[0044] Step 1: Use the RCF network to obtain the edge contour of the target in the original image X and obtain the edge information graph P;

[0045] Step 2: Train the generator;

[0046] Keep the parameters of the discriminator fixed, and transform the edge information graph P and the generated image G(X att ) is input into the discriminator for training, where X att =λX+(1-λ)P represents the edge feature enhancement map, λ represents the fusion weight;

[0047] Step 3: Train the discriminator;

[0048] Keep the parameters of the generator fixed, fuse the original image X and the edge information map P, and then merge the fused label image Y with the generated image G(X att ) input the discriminator for training;

[0049] Step 4: Use the trained generator to generate the target segmentation image.

[0050] Embodiment 2

[0051] This embodiment provides an occlusion-aware segmentation method based on the RCF-Pix2Pix deep network, which combines the RCF edge extraction network with Pix2Pix and uses different conditional information to allow the discriminator to make authenticity judgments, so as to enhance the focus on the edge area. At the same time, the edge information and the overall contour are used as guidance, so that Pix2Pix can complete the segmentation task relatively accurately even when the chip contour is disturbed. Specifically, the following steps are included:

[0052] Step 1: Use the RCF network to obtain the edge contour of the target in the original image X and obtain the edge information graph P;

[0053] like Figure 1 As shown in the figure, in order to obtain edge information in the image, an RCF network is cascaded before the generator to first extract the edge contour of the target in the image. The RCF model takes advantage of the deep convolutional neural network in extracting rich features. Through multi-level feature extraction, it can capture fine-grained information in the image, thereby improving the accuracy of edge detection. Compared with traditional edge detection methods, the RCF model performs better under complex backgrounds and noise interference. First, the edge information map P of the input image X is extracted by RCF, as shown in formula (1). Then the extracted edge information map P is fused with the original image X to enhance the edge features of the image, and then the fused image Xatt is delivered to the generator for segmentation. Such a processing flow can make the edge features of the target in the image clearer and more obvious, and enhance the edge features in the original image. Formula (2) is the fusion method of the edge feature enhancement map Xatt.

[0054] P=RCF(X) (1)

[0055] X att =λX+(1-λ)P (2)

[0056] Wherein, λ is the fusion weight, and in this embodiment, it is set to 0.9 for the best effect through experiments.

[0057] Step 2: Train the generator;

[0058] During the generator training process, a discriminator with unchanged parameters is used to generate the image G(X att) to judge the authenticity of the generated image, thereby improving the segmentation ability of the generator. In this process, this embodiment changes the conditional information from the original image (X) to the edge map (P) extracted by RCF. Using these edge maps as the conditional information of the discriminator provides richer local feature information, which helps the generator better understand the structure and details of the original image.

[0059] When training the generator, this embodiment uses the least squares conditional GAN ​​loss, MSE-SSIM-L1 loss, and edge loss for loss calculation. These loss functions can help the generator generate more stable segmented images during training, and also have better segmentation accuracy.

[0060] Compared with the traditional GAN ​​loss function, the least squares conditional GAN ​​pays more attention to the details of the generated image. Through the least squares loss, the generator can better learn and retain the structure and detail information of the input conditional image (such as edge image). This helps to generate more realistic and detailed image results. The calculation is shown in formula (3), where T norm It is an N*N matrix filled with 1. The size of N is related to the setting in PatchGan. This embodiment uses the default size of 70*70.

[0061]

[0062] This embodiment uses the MS-SSIM-L1 loss to replace the original L1 loss. The original Pix2Pix network uses the L1 loss function to calculate the loss value between the label map and the generated map. However, in the presence of noise or outliers, it is very sensitive to these noise or outliers. Since the L1 loss is calculated based on the absolute difference, it will give a large penalty even if there are extreme outliers, causing the model to be overly sensitive to these outliers, thereby affecting the robustness and generalization ability of the model.

[0063] MS-SSIM-L1 loss not only considers the pixel-level difference (L1 loss), but also introduces the structural similarity index, which can better measure the structural features of the image, so that when calculating the loss, more attention is paid to the structure, texture and other features of the image, rather than just the pixel-level difference. This can improve the model while maintaining the image structure while reducing the distortion of details, which helps to improve the quality of the generated image and the segmentation accuracy. As shown in formula (4).

[0064]

[0065] Where, L MS-SSIM is the MS-SSIM combined loss, is the Gaussian filter with standard deviation of the Mth layer in the pyramid structure, L l1 is the L1 loss function, Represents the weight of MS-SSIM loss, which is set to 0.84 based on experience 3 .

[0066] In the generative adversarial network, the generated image is often prone to blur and confusion, especially at the target boundary. By introducing the edge loss function, the blur and confusion of the generated image can be effectively suppressed, making the segmentation result clearer and more accurate. Therefore, this embodiment adds an edge loss function, by extracting the edges of the generated image and the label image, and only calculating the L1 loss at the edge, as shown in formula (5), the edges of the generated image and the label image are extracted by RCF, and then calculated:

[0067]

[0068] The edge loss function can provide additional structural constraints, which can help the model better understand and distinguish the boundaries between objects, making the generated images more consistent with the contours and edges of objects in the real world. This helps avoid unnatural or blurred edges in the generated images, thereby improving the accuracy of segmentation.

[0069] In summary, the overall loss function of the generator in this embodiment is shown in formula (6).

[0070]

[0071] Step 3: Train the discriminator;

[0072] During the training of the discriminator, the parameters of the generator remain fixed. This process involves training the label map Y and the generated map G(X att ) is input into the discriminator to improve the discriminator's ability to judge true and false result images. att As a condition, the edge information in the image can be enhanced. The edge result map can provide additional information about the image structure and boundaries. The original image may contain some blurred areas or parts with unclear details, and the edge result map can provide accurate edge information. By fusing the original image and the edge result map, the discriminator can more comprehensively understand the content and boundary features of the image. This method can improve the performance of non-modal segmentation, making it more accurate in identifying the occluded and complete parts of the object.

[0073] When the discriminator is trained, different T-norm labels are used for the real image and the generated image to calculate the least squares loss. When judging the real image, the matrix T filled with 0 is used. 0 , when judging the true graph, use the matrix T filled with 1 norm , the size of both matrices is 70*70, and the calculation is shown in formula (7).

[0074]

[0075] Where n represents the number of blocks after the image is divided, N, T 0 represents an N*N matrix filled with 0, T norm represents an N*N matrix filled with 1, D(Y) represents the result image after the real image passes through the discriminator, D(G(X att )) represents the result image after the fake image produced by the generator passes through the discriminator.

[0076] Step 4: Use the trained generator to generate the target segmentation image.

[0077] In order to evaluate the performance of the RCF-Pix2Pix algorithm, this example uses a dataset of SOT-323 chip images for comparative experiments. First, the good images are marked, and then defects such as cracks, dirt, and damage are added using Photoshop. In order to further diversify the defect types, irregular black areas are randomly generated in the good images. The image size is 512x512 pixels, with 1343 images in the training set and 310 images in the test set.

[0078] The experimental environment runs on Windows 10 operating system, equipped with Intel Core i9-12900k CPU and GeForce RTX 3090 GPU. The development environment includes Python 3.8, Pytorch 1.10.0, CUDA 11.3, Pycharm IDE and Visual Studio 2022 IDE. The training parameters of the segmentation model are set to 100 epochs and the batch size is 1. The optimizer used is Adam. The initial learning rate is set to 1e-5 and the momentum is 0.5.

[0079] like Figure 2 As shown in the figure, the original Pix2Pix model showed significant fluctuations during training and low edge IOU values. After replacing the input image with the fused image, the fluctuation range was slightly reduced, but still not ideal. By adjusting the conditional information in the discriminator, the training process became more stable, and both IOU indicators improved significantly. The use of a new loss function further stabilized the quality of the segmented image and improved the segmentation accuracy. Changing the input conditions of the network made it more focused on the segmentation and restoration of the target edge area. Compared with the original network, the new network discarded the redundant information in the image, making the target more obvious and clear.

[0080] Table 1 lists the comparative experimental results of the original network and the improved module network:

[0081] Table 1 Comparative experiments of images under different input and conditions

[0082]

[0083] It can be seen that when the input image in the segmentation module is changed to a fused image, the accuracy is slightly improved. In addition, after changing the discrimination condition, the accuracy is significantly improved, especially in terms of boundary IoU, which increases by nearly 7.5%. Under the guidance of edge information, the network significantly improves the segmentation accuracy of normal samples and the recovery accuracy of abnormal samples.

[0084] Table 2 shows the comparative experimental results of using different loss functions combined with GAN loss in the process of training the generator:

[0085] Table 2 Using different loss functions

[0086]

[0087] There is a lot of noise in the original Pix2Pix generated image, and the stability of the generated image is not good. In order to solve this problem, the present invention adjusts and optimizes the network architecture and loss function. The results show that changing the loss function can not only stabilize the quality of the generated image, but also further improve the accuracy of non-modal segmentation. The modification of the loss function makes the network more sensitive to the edge of the target, so that the edge area can be accurately segmented and the occluded part can be successfully repaired. The research results show that the final segmentation result image generated by RCF-Pix2Pix has the best image quality, and only needs to be binarized to complete the binarization post-processing.

[0088] The segmentation capabilities of U-net and DeepLabV3

[40] are also compared on this dataset, as shown in Table 3:

[0089] Table 3 Comparison of chip segmentation results using different networks

[0090]

[0091] The results show that U-net has an mIoU of 97.88% and a Boundary IoU of 77.44% on this dataset, while DeepLabV3+ has an mIoU of 94.52% and a Boundary IoU of 58.60%. It can be observed that compared with the original Pix2Pix and DeepLabV3+, U-net performs better in segmenting occlusion boundaries, but the segmentation effect of RCF-Pix2Pix of the present invention is the best.

[0092] Figure 3The comparison of segmentation results of different networks is shown. It can be seen from the figure that when the segmentation target is occluded, the target segmentation method based on the RCF-Pix2Pix network of the present invention has clearer and smoother edges, which proves that the present invention effectively improves the segmentation accuracy of the edge of the occluded area, thereby improving the target segmentation accuracy.

[0093] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An occlusion-aware target segmentation method, characterized in that: The method comprises: Step 1: Use the RCF network to obtain the edge contour of the target in the original image X and obtain the edge information graph P; Step 2: Train the generator; Keep the parameters of the discriminator fixed, and transform the edge information graph P and the generated image G(X att ) is input into the discriminator for training, where X att =λX+(1-λ)P represents the edge feature enhancement map, λ represents the fusion weight; Step 3: Train the discriminator; Keep the parameters of the generator fixed, fuse the original image X and the edge information map P, and then merge the fused label image Y with the generated image G(X att ) input the discriminator for training; Step 4: Use the trained generator to generate the target segmentation image.

2. The occlusion-aware target segmentation method according to claim 1, characterized in that: The loss function of the training generator in step 2 is: in, represents the least squares conditional GAN ​​loss, represents the MS-SSIM-L1 loss, Represents marginal loss.

3. The occlusion-aware target segmentation method according to claim 1, characterized in that: The loss function of the step 3 training discriminator is: Where n represents the number of blocks after the image is divided, T0 represents an N*N matrix filled with 0, and T norm represents an N*N matrix filled with 1, D(Y) represents the result image after the real image passes through the discriminator, D(G(X att )) represents the result image after the fake image produced by the generator passes through the discriminator.

4. The occlusion-aware target segmentation method according to claim 1, characterized in that: The fusion weight λ is set to 0.

9.

5. An occlusion-aware target segmentation system, characterized in that: The system comprises: The edge extraction module is configured to obtain the edge contour of the target in the original image X by using the RCF network to obtain the edge information map P; The image generation module is configured to keep the parameters of the discriminator fixed and transform the edge information map P and the generated image G(X att ) input to train the discriminator, where X att =λX+(1-λ)P represents the edge feature enhancement map, λ represents the fusion weight; The image identification module is configured to keep the parameters of the generator fixed, fuse the original image X and the edge information map P, and then fuse the fused label image Y with the generated image G(X att ) input to train the discriminator; The object segmentation module is configured to generate an object segmentation image using the trained generator.

6. The occlusion-aware target segmentation system according to claim 5, characterized in that: The loss function of the image generation module training generator is: in, represents the least squares conditional GAN ​​loss, represents the MS-SSIM-L1 loss, Represents marginal loss.

7. The occlusion-aware target segmentation system according to claim 5, characterized in that: The loss function of the image identification module training discriminator is: Where n represents the number of blocks after the image is divided, T0 represents an N*N matrix filled with 0, and T norm represents an N*N matrix filled with 1, D(Y) represents the result image after the real image passes through the discriminator, D(G(X att )) represents the result image after the fake image produced by the generator passes through the discriminator.

8. The occlusion-aware target segmentation system according to claim 5, characterized in that: The fusion weight λ is set to 0.

9.

9. An occlusion-aware target segmentation device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the occlusion-aware target segmentation method as described in any one of claims 1 to 4 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the occlusion-aware target segmentation method according to any one of claims 1 to 4 is implemented.

Images