Occlusion-Aware Segmentation Method Based on the RCF-Pix2Pix Deep Network

Abstract

The present invention discloses an occlusion-aware segmentation method based on the RCF-Pix2Pix deep network, belonging to the technical field of digital image processing. The method includes: obtaining the edge contour of the target in the original image X by using the RCF network to obtain the edge information map P; training the generator and the discriminator respectively by using an improved loss function; and generating the target segmentation image by using the trained generator. The present invention combines the RCF edge extraction network with Pix2Pix, and uses different conditional information to let the discriminator judge the authenticity, so as to enhance the attention in the edge region; at the same time, combining the edge information and the overall contour as a guide, so that Pix2Pix can relatively accurately complete the segmentation task even when the target contour is interfered; the experimental results prove that compared with the existing segmentation methods, the present invention shows higher segmentation accuracy in segmenting the occlusion boundary.

Description

Technical Field

[0001] The present invention relates to an occlusion-aware segmentation method based on the RCF-Pix2Pix deep network, and belongs to the technical field of digital image processing. Background Art

[0002] In recent years, vision recognition tasks such as image classification, object detection, edge detection, and semantic segmentation have achieved relatively successful results, which benefit from the development of large-scale datasets and deep learning technologies. The annotation process of most datasets does not consider the need to indicate occlusion areas, so they cannot be used for non-modal perception and cannot well predict the shape contours of occluded areas. However, being able to predict the complete contour of an object is crucial for understanding the scene in some fields. For example, in the field of agricultural production, by successfully predicting the shape of fruits occluded by leaves under different lighting conditions, the stability and accuracy of the manipulator performing fruit picking tasks can be improved. Similarly, in autonomous driving technology, enabling the vehicle to infer the complete shapes of occluded vehicles and pedestrians within the line of sight can significantly reduce the risk of collisions. In addition, in industrial production scenarios, since products may have overlaps, debris residues, or shape deformations due to defects, being able to predict the original contour of the product is crucial for extracting target surface information and ensuring precise and stable grasping by the manipulator.

[0003] Currently, there are some methods for occluded object segmentation by improving convolutional neural networks. However, these methods generally suffer from problems such as complex and time-consuming training, poor robustness in multi-scale object segmentation, and large dataset requirements. In addition to convolutional neural networks, generative adversarial networks (GANs) have also been used in image segmentation tasks. Mazumdar, H. et al. used the GAN-based image transformation framework Pix2Pix to develop an efficient and reliable polyp segmentation method (Mazumdar, H., Chakraborty, C., Sathvik, M., Jayakumar, P., Kaushik, A.: Optimizing pix2pix gan with attention mechanisms for ai-driven polyp segmentation in iomt-enabled smart healthcare, 2023); Chen, Z. et al. compared the trunk segmentation capabilities of Pix2Pix, U-Net, and DeepLabv3 in images occluded by leaves, and the results showed that Pix2Pix achieved the highest segmentation accuracy (Chen, Z., Ting, D., Newbury, R., Chen, C.: Semantic segmentation for partially occluded apple trees based on deep learning, 2021). However, Pix2Pix has certain instability in segmenting edge regions. Especially in the case of large-scale contour loss, there will be a significant difference between the edge of the segmentation result and the actual edge. Summary of the Invention

[0004] In order to enhance the segmentation accuracy of occluded regions, the present invention provides an occlusion-aware segmentation method based on the RCF-Pix2Pix deep network. The technical solution is as follows:

[0005] The occlusion-aware object segmentation method of the present invention includes:

[0006] Step 1: Use the RCF network to obtain the edge contour of the target in the original image X, and obtain the edge information map P;

[0007] Step 2: Train the generator;

[0008] Keep the parameters of the discriminator fixed, and input the edge information map P and the generated image G(X att ) into the discriminator for training, where X att = λX+(1 - λ)P represents the edge feature enhanced map, and λ represents the fusion weight;

[0009] Step 3: Train the discriminator;

[0010] Keep the parameters of the generator fixed, fuse the original image X and the edge information map P, and then input the fused labeled image Y and the generated image G(X att ) into the discriminator for training;

[0011] Step 4: Use the trained generator to generate the target segmentation image.

[0012] Optionally, the loss function for training the generator in Step 2 is:

[0013]

[0014] where represents the least squares conditional GAN loss, represents the MS-SSIM-L1 loss, represents the edge loss.

[0015] Optionally, the loss function for training the discriminator in Step 3 is:

[0016]

[0017] where n represents the number of blocks N after image partitioning, T0 represents an N*N matrix filled with 0s, T norm represents an N*N matrix filled with 1s, D(Y) represents the result map of the real image after passing through the discriminator, and D(G(X att )) represents the result map of the fake image produced by the generator after passing through the discriminator.

[0018] Optionally, the fusion weight λ is taken as 0.9.

[0019] The present invention provides an occlusion-aware target segmentation system, including:

[0020] An edge extraction module configured to use the RCF network to obtain the edge contour of the target in the original image X and obtain the edge information map P;

[0021] An image generation module configured to keep the parameters of the discriminator fixed and input the edge information map P and the generated image G(X att ) for training the discriminator, where X att =λX+(1-λ)P represents the edge feature enhanced map, and λ represents the fusion weight;

[0022] An image discrimination module configured to keep the parameters of the generator fixed, fuse the original image X and the edge information map P, and then input the fused labeled image Y and the generated image G(X att)Input for discriminator training;

[0023] A target segmentation module, configured to generate a target segmentation image using the trained generator.

[0024] Optionally, the loss function for the image generation module to train the generator is:

[0025]

[0026] Wherein, Represents the least squares conditional GAN loss, Represents the MS-SSIM-L1 loss, Represents the edge loss.

[0027] Optionally, the loss function for the image discrimination module to train the discriminator is:

[0028]

[0029] Wherein, n represents the number of blocks N after image division, T0 represents an N*N matrix filled with 0s, T norm Represents an N*N matrix filled with 1s, D(Y) represents the result image after the real image passes through the discriminator, D(G(X att )) represents the result image after the fake image produced by the generator passes through the discriminator.

[0030] Optionally, the fusion weight λ takes 0.9.

[0031] The present invention provides an occlusion-aware target segmentation device, including a memory and a processor;

[0032] The memory is used to store computer programs;

[0033] The processor is configured to, when executing the computer program, implement the occlusion-aware target segmentation method as described in any one of the above.

[0034] The present invention provides a computer-readable storage medium, characterized in that a computer program is stored on the storage medium, and when the computer program is executed by a processor, the occlusion-aware target segmentation method as described in any one of the above is implemented.

[0035] The beneficial effects of the present invention are:

[0036] The present invention designs an edge-enhanced Pix2Pix generative adversarial network RCF-Pix2Pix. By combining the RCF edge extraction network with Pix2Pix and using different conditional information for the discriminator to judge authenticity, the attention in the edge region is enhanced. At the same time, by combining edge information and overall contours as guidance, Pix2Pix can relatively accurately complete the segmentation task even when the target contour is disturbed. Experimental results prove that compared with existing segmentation methods, the present invention shows higher segmentation accuracy in segmenting occluded boundaries. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0038] Figure 1 It is the network structure diagram of the RCF-Pix2Pix of the present invention.

[0039] Figure 2 It is the training loss curve graph after gradually optimizing the network in the second embodiment of the present invention.

[0040] Figure 3 It is the effect visualization diagram of the present invention and other existing image segmentation schemes. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] To make the purpose, technical solutions and advantages of the present invention clearer, the following will further describe the embodiments of the present invention in detail in conjunction with the drawings.

[0042] Embodiment 1:

[0043] This embodiment provides an occlusion-aware target segmentation method, including:

[0044] Step 1: Use the RCF network to obtain the edge contour of the target in the original image X, and obtain the edge information map P;

[0045] Step 2: Train the generator;

[0046] Keep the parameters of the discriminator fixed, and input the edge information map P and the generated image G(X att ) into the discriminator for training, where X att = λX+(1 - λ)P represents the edge feature enhanced map, and λ represents the fusion weight;

[0047] Step 3: Train the discriminator;

[0048] Keep the parameters of the generator fixed, fuse the original image X and the edge information map P, and then input the fused label image Y and the generated image G(X att ) into the discriminator for training;

[0049] Step 4: Use the trained generator to generate the target segmentation image.

[0050] Example 2

[0051] This embodiment provides an occlusion-aware segmentation method based on the RCF-Pix2Pix deep network. By combining the RCF edge extraction network with Pix2Pix and using different conditional information for the discriminator to judge authenticity, the attention in the edge region is enhanced. At the same time, combining edge information and overall contour as guidance enables Pix2Pix to relatively accurately complete the segmentation task even when the chip contour is interfered. The specific steps are as follows:

[0052] Step 1: Use the RCF network to obtain the edge contour of the target in the original image X and obtain the edge information map P;

[0053] As Figure 1 shown, in order to obtain the edge information in the image, an RCF network is cascaded in front of the generator to first extract the edge contour of the target in the image. The RCF model takes advantage of the deep convolutional neural network in extracting rich features. Through multi-level feature extraction, it can capture the fine-grained information in the image, thereby improving the accuracy of edge detection. Compared with traditional edge detection methods, the RCF model performs better under complex backgrounds and noise interference. First, the edge information map P of the input image X is extracted by RCF as shown in Equation (1). Then the extracted edge information map P is fused with the original image X to enhance the edge features of the image, and the fused image Xatt is delivered to the generator for segmentation. Such a processing flow can make the edge features of the target in the image clearer and more obvious, enhancing the edge features in the original image. Equation (2) is the fusion method of the edge feature enhanced map Xatt.

[0054] P = RCF(X) (1)

[0055] X att = λX + (1 - λ)P (2)

[0056] where λ is the fusion weight, which is set to 0.9 through experiments for the best effect in this embodiment.

[0057] Step 2: Train the generator;

[0058] During the training process of the generator, a discriminator with unchanged parameters is used to judge the generated image G(X att) Score to judge the authenticity of the generated image, thereby improving the segmentation ability of the generator. In this process, in this embodiment, the conditional information is changed from the original image (X) to the edge map (P) extracted by RCF. Using these edge maps as the conditional information of the discriminator provides richer local feature information, which helps the generator better understand the structure and details of the original image.

[0059] During the training of the generator, in this embodiment, the least - square conditional GAN loss, MSE - SSIM - L1 loss, and edge loss are used for loss calculation. These loss functions can help the generator generate more stable segmentation images during training and also have better segmentation accuracy.

[0060] The least - square conditional GAN pays more attention to the details of the generated image compared to the traditional GAN loss function. Through the least - square loss, the generator can better learn and retain the structural and detailed information of the input conditional image (such as the edge image). This helps to generate more realistic and detailed image results. The calculation is as shown in Equation (3), where T norm is an N * N matrix filled with 1s, and the size of N is related to the setting in PatchGan. In this embodiment, the default size of 70 * 70 is used.

[0061]

[0062] In this embodiment, the MS - SSIM - L1 loss is used to replace the original L1 loss. In the original Pix2Pix network, the L1 loss function is used to calculate the loss value between the label map and the generated map. However, in the presence of noise or outliers, it is very sensitive to these noise or outliers. Since the L1 loss is calculated based on the absolute difference, even in the presence of extreme outliers, it will give a large penalty, causing the model to be overly sensitive to these outliers, thus affecting the robustness and generalization ability of the model.

[0063] The MS - SSIM - L1 loss not only considers the pixel - level difference (L1 loss), but also introduces the structural similarity index, which can better measure the structural features of the image. Thus, when calculating the loss, it pays more attention to features such as the structure and texture of the image, rather than just the pixel - level difference. This can improve the model's ability to reduce detail distortion while maintaining the image structure, helping to improve the quality and segmentation accuracy of the generated map. As shown in Equation (4).

[0064]

[0065] In the formula, L MS-SSIM is the MS - SSIM combined loss, is the Gaussian filter with a standard deviation at the M - th layer of the pyramid structure, and L l1 is the L1 loss function. Represents the weight of the MS - SSIM loss, which is empirically set to 0.84 3 。

[0066] In the generative adversarial network, the generated images often tend to be blurred and confused, especially at the target boundary parts. By introducing an edge loss function, the blurring and confusion phenomena of the generated images can be effectively suppressed, thus making the segmentation results clearer and more accurate. Therefore, in this embodiment, an edge loss function is added. By extracting the edges of the generated image and the label image, only the L1 loss at the edges is calculated, as shown in Equation (5). The edges of the generated image and the label image are extracted by RCF and then calculated:

[0067]

[0068] This edge loss function can provide additional structural constraints, which can help the model better understand and distinguish the boundaries between targets, making the generated images more conform to the contours and edges of objects in the real world. This helps to avoid unnatural or blurred edges in the generated images, thereby improving the segmentation accuracy.

[0069] In summary, the overall loss function of the generator in this embodiment is as shown in Equation (6).

[0070]

[0071] Step 3: Train the discriminator;

[0072] During the training process of the discriminator, the parameters of the generator remain fixed. This process involves inputting the label image Y and the generated image G(X att ) into the discriminator to enhance the discriminator's ability to judge true and false result images. At the same time, using X att as a condition can enhance the edge information in the image. The edge result image can provide additional information about the image structure and boundaries. The original image may contain some blurred regions or parts with unclear details, while the edge result image can provide precise edge information. By fusing the original image and the edge result image, the discriminator can more comprehensively understand the content and boundary features of the image. This method can improve the performance of non - modal segmentation, enabling it to more accurately identify the occluded and complete parts of objects.

[0073] When training the discriminator, different T - norm labels are used for the least - square loss calculation of the real image and the generated image. When judging the real image, the matrix T0 filled with 0 is used, and when judging the generated image, the matrix T norm is used. Both matrices are of size 70 * 70, and the calculation is as shown in Equation (7).

[0074]

[0075] Among them, n represents the number of blocks N after image division, T0 represents an N*N matrix filled with 0s, and T norm represents an N*N matrix filled with 1s, D(Y) represents the result image after the real image passes through the discriminator, and D(G(X att )) represents the result image after the fake image generated by the generator passes through the discriminator.

[0076] Step 4: Use the trained generator to generate the target segmentation image.

[0077] To evaluate the performance of the RCF-Pix2Pix algorithm, this embodiment uses a dataset of SOT-323 chip images for comparative experiments. First, the good-quality images are marked, and then defects such as fragmentation, dirt, and damage are added using Photoshop. To further diversify the defect types, irregular black areas are randomly generated in the good-quality images. The image size is 512x512 pixels, with 1343 images in the training set and 310 images in the test set.

[0078] The experimental environment runs on the Windows 10 operating system, equipped with an Intel Core i9-12900k CPU and a GeForce RTX 3090 GPU. The development environment includes Python 3.8, Pytorch 1.10.0, CUDA 11.3, Pycharm IDE, and Visual Studio 2022 IDE. The training parameters of the segmentation model are set to 100 epochs, and the batch size is 1. The optimizer used is Adam. The initial learning rate is set to 1e-5, and the momentum is 0.5.

[0079] As Figure 2 shown, the original Pix2Pix model shows significant fluctuations during training, with a low edge IOU value. After replacing the input image with the fused image, the fluctuation range is slightly reduced, but it is still not ideal. By adjusting the conditional information in the discriminator, the training process becomes more stable, and both IOU metrics are significantly improved. Using the new loss function further stabilizes the quality of the segmented image and improves the segmentation accuracy. Changing the input conditions of the network makes it more focused on the segmentation and restoration of the target edge region. Compared with the original network, the new network discards the redundant information in the image, making the target more obvious and clear.

[0080] Table 1 lists the comparative experimental results of the original network and the improved module network:

[0081] Table 1 Comparative experiments of images under different inputs and conditions

[0082]

[0083] It can be seen that when the input image in the segmentation module is changed to the fused image, the accuracy is slightly improved. In addition, after changing the discrimination condition, the accuracy is significantly improved, especially in terms of the boundary IoU, which increases by nearly 7.5%. Under the guidance of edge information, the network significantly improves the segmentation accuracy of normal samples and the recovery accuracy of abnormal samples.

[0084] Table 2 shows the comparative experimental results of using different loss functions in combination with the GAN loss during the training of the generator:

[0085] Table 2 Using different loss functions

[0086]

[0087] In the images generated by the original Pix2Pix, there is a lot of noise and the stability of the generated images is not good. To solve this problem, the present invention adjusts and optimizes the network architecture and loss function. The results show that changing the loss function can not only stabilize the quality of the generated images, but also further improve the accuracy of non-modal segmentation. The modification of the loss function makes the network more sensitive to the target edges, so that it can accurately segment the edge regions and successfully repair the occluded parts. The research results show that the final segmentation result images generated by RCF-Pix2Pix have the best image quality and only need to be binarized to complete the post-binarization processing.

[0088] The segmentation capabilities of U-net and DeepLabV3

[40] were also compared on this dataset, as shown in Table 3:

[0089] Table 3 Comparison of the results of chip segmentation using different networks

[0090]

[0091] The results show that the mIoU of U-net on this dataset is 97.88% and the Boundary IoU is 77.44%, while the mIoU of DeepLabV3+ is 94.52% and the Boundary IoU is 58.60%. It can be observed that compared with the original Pix2Pix and DeepLabV3+, U-net performs better in segmenting occluded boundaries, but the segmentation effect of the RCF-Pix2Pix of the present invention is the best.

[0092] Figure 3The comparison of the segmentation results of different networks is shown. It can be seen from the figure that when the segmentation target is occluded, the edge of the target segmentation method based on the RCF-Pix2Pix network of the present invention is clearer and smoother, proving that the present invention effectively improves the segmentation accuracy of the edge of the occluded area, thereby improving the target segmentation accuracy.

[0093] Some steps in the embodiments of the present invention can be implemented by software, and the corresponding software program can be stored in a readable storage medium, such as an optical disc or a hard disk, etc.

[0094] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An occlusion-aware object segmentation method, characterized in that, The method includes: Step 1: Use the RCF network to obtain the original image X and obtain the edge contour of the target in it to get the edge information map P ; Step 2: Train the generator; Keep the parameters of the discriminator fixed, and input the edge information map P and the generated image into the discriminator for training, where represents the edge feature enhancement map, represents the fusion weight; Step 3: Train the discriminator; Keep the parameters of the generator fixed and use the original image X and the edge information map P for fusion. Then input the fused labeled image Y and the generated image into the discriminator for training; Step 4: Use the trained generator to generate the target segmentation image; The loss function for training the generator in the said Step 2 is: Among them, represents the least squares conditional GAN loss, represents the MS-SSIM-L1 loss, represents the edge loss.

2. The occlusion-aware object segmentation method according to claim 1, wherein The loss function for training the discriminator in the said Step 3 is: Among them, n represents the number of blocks N after image partitioning, represents an N*N matrix filled with 0s, represents an N*N matrix filled with 1s, represents the result image after the real image passes through the discriminator, represents the result image after the fake image generated by the generator passes through the discriminator.

3. The occlusion-aware target segmentation method according to claim 1, wherein The fusion weight Take 0.

9.

4. An occlusion-aware object segmentation system, characterized in that, The system includes: An edge extraction module, configured to obtain an original image by using an RCF network X and obtain an edge contour of a target therein to obtain an edge information map P ; An image generation module, configured to keep the parameters of the discriminator fixed and input the edge information map P and the generated image for training the discriminator, where represents an edge feature enhancement map, represents a fusion weight; The image discrimination module is configured to keep the parameters of the generator fixed and fuse the original image X and the edge information map P to obtain a fused labeled image Y, which is then input together with the generated image for training the discriminator; A target segmentation module configured to use the trained generator to generate the target segmentation image.

5. The occlusion-aware object segmentation system according to claim 4, wherein The loss function for the image generation module to train the generator is: Among them, represents the least squares conditional GAN loss, represents the MS-SSIM-L1 loss, represents the edge loss.

6. The occlusion-aware object segmentation system according to claim 4, wherein The loss function for the image discrimination module to train the discriminator is: Among them, n represents the number of blocks N after image division, represents an N*N matrix filled with 0s, represents an N*N matrix filled with 1s, represents the result image after the real image passes through the discriminator, represents the result image after the fake image generated by the generator passes through the discriminator.

7. The occlusion-aware object segmentation system according to claim 4, wherein The fusion weight is taken as 0.

9.

8. An occlusion-aware object segmentation device, characterized in that, It includes a memory and a processor; The memory is used for storing a computer program; The processor is used for, when executing the computer program, implementing the occlusion-aware target segmentation method according to any one of claims 1 to 3.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the storage medium, and when the computer program is executed by the processor, the occlusion-aware target segmentation method according to any one of claims 1 to 3 is implemented.

Images