A method and apparatus for transparent object segmentation assisted by user interaction information

By generating Gaussian heatmaps through user interaction and combining them with a deep learning network, the problems of boundary blurring and reflection effects in transparent object segmentation are solved, achieving high-precision transparent object segmentation.

CN118522014BActive Publication Date: 2026-11-17GUANGXI UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410631264.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-21
Publication Date
2026-11-17
Estimated Expiration
2044-05-21

AI Technical Summary

Technical Problem

Existing image segmentation methods struggle to accurately segment transparent objects, especially when boundaries are blurred or when reflection and refraction are present. Traditional methods suffer performance degradation, and sensor-based methods are susceptible to interference.

Method used

A user-interactive information-assisted approach is adopted to generate Gaussian heatmaps through multi-point interaction. Combined with ResNet50 backbone network and aggregated dilated spatial pooling, imbalanced loss is used for feature extraction and segmentation. The generated Gaussian heatmap is then stitched with the original image to refine the feature map and balance the foreground and background.

Benefits of technology

It significantly improves the accuracy and stability of transparent object segmentation, especially in images with high transparency and irregular boundaries, and solves the mesh effect and class imbalance problem of empty space pooling.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118522014B_ABST
    Figure CN118522014B_ABST
Patent Text Reader

Abstract

This invention discloses a method and apparatus for segmenting transparent objects with user interaction information assistance, comprising the following steps: converting auxiliary information generated by multi-point interaction between the user and transparent objects in the image to be segmented into a Gaussian heatmap through Gaussian transform; combining the converted Gaussian heatmap with the original image to form a 5-channel input; extracting initial features through a backbone network, and then further refining the feature map by combining multi-scale information and contextual information through aggregated dilated spatial pooling; balancing the foreground and background classes during training through class imbalance loss; separating the foreground and background of the transparent object image based on the refined and balanced feature map, and outputting the final segmented image. This invention is applicable to the segmentation of complex images with high transparency, irregular boundaries, or multiple transparent targets, achieving good segmentation accuracy and results using only a minimal amount of user interaction information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of image semantic segmentation technology, specifically to a method and apparatus for segmenting transparent objects with the assistance of user interaction information. Background Technology

[0002] With the development of machine vision and image processing technologies, image semantic segmentation has been widely applied in many fields, such as autonomous driving, medical image analysis, and robot navigation. The goal of image semantic segmentation is to assign each pixel in an image to a predefined category, thereby achieving an understanding of the image content. However, existing semantic segmentation methods often face significant challenges when it comes to segmenting images containing transparent objects. Due to their refractive and reflective properties, transparent objects make the information of objects behind them visible in the image, resulting in blurred boundaries between the transparent object and the background, making accurate segmentation difficult. Furthermore, the color and texture of transparent objects may be similar to the background, making it difficult for traditional color- and texture-based segmentation methods to effectively distinguish them.

[0003] Existing image segmentation methods, such as threshold-based, region-based, and edge-based segmentation, while providing reasonable segmentation results in some cases, often suffer significant performance degradation when dealing with transparent objects. Furthermore, although some deep learning-based methods have made significant progress in image segmentation, they still have limitations when handling the complex light propagation effects of transparent objects.

[0004] Current mainstream transparent object segmentation methods often rely on sensors such as sonar or radar to perceive information from the surrounding environment and thus assist in segmenting transparent objects. However, due to the reflective, refractive, and easily penetrable characteristics of transparent objects, they can easily interfere with the information acquired by the sensors. Therefore, from a computer vision perspective, designing a reliable and accurate transparent object segmentation method can not only save costs but also overcome the shortcomings of sensor-based methods, achieving more stable and accurate segmentation results.

[0005] In recent years, a method has emerged that utilizes minimal user interaction as supplementary information to assist image segmentation. Interactive image segmentation aims to obtain high-quality pixel-level annotations using minimal user interaction. Common interaction methods include clicking, drawing, and bounding boxes, with clicking being the simplest and most prevalent. These human interactions can provide auxiliary information for image segmentation, thereby improving segmentation results. A common approach to combining interactive image segmentation with deep learning involves adding positive interaction points to the image to be segmented to label the foreground and negative interaction points to label the background. Then, the interaction information is converted into a Euclidean distance map, which is used as an additional input channel into a fine-tuned neural network.

[0006] In the process of realizing this invention, the inventors discovered at least the following problems in the prior art:

[0007] (1) Transparent objects lack texture, are easy to blend with the background, and are easily affected by light refraction and reflection, making it difficult to accurately separate their foreground and background in image segmentation.

[0008] (2) Hollow space pooling has a grid effect, which can easily lead to feature loss and lack of correlation of long-distance information.

[0009] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0010] The purpose of this invention is to provide a transparent object segmentation method and apparatus assisted by user interaction information, thereby achieving accurate semantic segmentation of transparent objects. In particular, for complex images with high transparency of the target to be segmented, irregular boundaries, or multiple segmentation targets, the performance of this invention is more significant.

[0011] To achieve the above objectives, the present invention provides a transparent object segmentation method and apparatus assisted by user interaction information, comprising the following steps:

[0012] Step S1: Convert the auxiliary information generated by the user's multi-point interaction with transparent objects in the image to be segmented into a Gaussian heatmap through Gaussian transformation;

[0013] Step S2: Combine the converted Gaussian heatmap with the original image to form a 5-channel input;

[0014] Step S3: Extract initial features through the backbone network, and then further refine the feature map by combining multi-scale information and contextual information through aggregated dilated spatial pooling to obtain the refined feature map;

[0015] Step S4: Balance the foreground and background classes during training using class imbalance loss;

[0016] Step S5: Separate the foreground and background of the transparent object image based on the refined and balanced feature map, and output the final segmented image.

[0017] Furthermore, the multi-point interaction described in step S1 includes four extreme points (leftmost, rightmost, topmost, and bottommost) of the object to be segmented, additional positive points, and negative points. The extreme points are used to mark the pixel region where the target to be segmented is located, the additional positive points are used to indicate the foreground, and the negative points are used to indicate the background. The Gaussian heatmap generated by the Gaussian transform is the same size as the original image, ensuring that its heatmap information is consistent with the original image.Figure 1 One-to-one correspondence; the four extreme points and positive points are on the same heatmap, which acts on the foreground class. The four extreme points provide spatial information of the object to be segmented to the segmentation network, while the positive points provide feature information that is difficult to capture due to reflection, high transparency, blurred boundaries, etc. The negative points are on another separate heatmap, which acts on the background class and provides feature information of the target interior that is difficult to identify as background to the segmentation network.

[0018] Furthermore, the 5-channel input in step S2 is generated by converting the user-provided extreme points, positive points (used to indicate the foreground), and negative points (used to indicate the background) interaction information into two heatmaps corresponding to the foreground and background after Gaussian transform encoding. These heatmaps are then stitched together with the R, G, and B channels of the original transparent object image.

[0019] Furthermore, in step S3, ResNet50 is used as the backbone network, and aggregated dilated spatial pooling is used to further refine the initial features. The process is as follows: after the input image is processed by the backbone network to extract the initial features, the network is divided into two branches: boundary flow and normal flow. The boundary flow generates boundary information through aggregated dilated spatial pooling and the decoder, while the normal flow, after aggregated dilated spatial pooling, uses the boundary information generated by the boundary flow as a guide in the boundary attention module to generate the refined feature map.

[0020] The aggregated void space pooling further refines the initial features by combining multi-scale information and contextual information, and its process is as follows:

[0021] Image features processed by pooling layers and dilated convolutions with different dilation rates are upsampled and then added to the original features processed by 1×1 convolution. The added features are then subjected to 3×3 convolution for the first feature aggregation. The aggregated features are then concatenated with the original features. The concatenated feature map is then processed by 1×1 convolution and added to the original features. Finally, the added features are subjected to another 1×1 convolution for the second feature fusion, resulting in aggregated multi-scale features.

[0022] Furthermore, step S4 employs class-imbalanced loss to balance the foreground and background classes, and its calculation method is as follows:

[0023]

[0024] l v =O*(G-O')-log(1+e (O-2(o*o')) )

[0025] Where N, N b N fThese represent the total number of pixels in the label, the number of background pixels, and the number of foreground pixels, respectively. O represents the predicted image output by the method, O' represents the image after binarizing O (pixels with pixel values ​​greater than or equal to 0 in O have their corresponding pixel values ​​in O' set to 1, otherwise set to 0), and G represents the label.

[0026] Based on the same inventive concept, embodiments of the present invention also provide a transparent object segmentation device assisted by user interaction information. This device comprises four modules, including:

[0027] The user interaction module generates auxiliary information by allowing users to interact with transparent objects in the image to be segmented at multiple points, and outputs an image with the user interaction information added.

[0028] Clicking the encoding module converts the user's multi-point interaction information into a Gaussian heatmap through Gaussian transformation, and combines it with the original image to generate a 5-channel input;

[0029] The feature extraction module performs preliminary feature extraction on the input through the backbone network, and then further refines the features through aggregated dilated spatial pooling.

[0030] The image segmentation module separates the foreground and background of the transparent object image based on the refined and balanced feature map, and outputs the final segmented image.

[0031] In one possible implementation, the multi-point interaction in the user interaction module includes:

[0032] The four extreme points (leftmost, rightmost, topmost, and bottommost) of the object to be segmented are used to provide spatial information of the object to be segmented for the segmentation network.

[0033] Positive points are used to indicate the foreground and provide segmentation networks with feature information that is difficult to capture due to factors such as reflection, high transparency, and blurred boundaries.

[0034] Negative points are used to indicate the background and provide segmentation networks with feature information about the interior of the target that is difficult to identify as background.

[0035] In one possible implementation, the Gaussian transform in the click encoding module uses a Gaussian distribution function (normal distribution function). The 5-channel input consists of user-provided extreme points, positive points (foreground indication), and negative points (for background indication) interactive information. After being encoded by Gaussian transform, these are converted into two heatmaps corresponding to the foreground and background, respectively, which are then spliced ​​together with the R, G, and B channels of the original transparent object image.

[0036] In one possible implementation, the feature extraction module uses ResNet50 as the backbone network and employs aggregated dilated spatial pooling to further refine the initial features. The process is as follows: after the input image is processed by the backbone network to obtain initial features, the network is divided into two branches: a boundary flow and a regular flow. The boundary flow generates boundary information through aggregated dilated spatial pooling and a decoder, while the regular flow, after aggregated dilated spatial pooling, uses the boundary information generated by the boundary flow as a guide in the boundary attention module to generate a refined feature map.

[0037] The aggregated void space pooling further refines the initial features by combining multi-scale information and contextual information, and its process is as follows:

[0038] Image features processed by pooling layers and dilated convolutions with different dilation rates are upsampled and then added to the original features processed by 1×1 convolution. The added features are then subjected to 3×3 convolution for the first feature aggregation. The aggregated features are then concatenated with the original features. The concatenated feature map is then processed by 1×1 convolution and added to the original features. Finally, the added features are subjected to another 1×1 convolution for the second feature fusion, resulting in aggregated multi-scale features.

[0039] The above four modules are encapsulated and deployed on a server platform to obtain a transparent object segmentation device assisted by user interaction information, which is used to achieve transparent object segmentation.

[0040] The present invention, by adopting the above-described technical solution, has the following beneficial effects:

[0041] (1) This invention significantly improves feature acquisition capability by using a small amount of user interaction as auxiliary information. The effect of this method is particularly significant when processing complex images with high transparency, irregular boundaries, or multiple segmentation targets.

[0042] (2) The aggregated void space pooling proposed in this invention can solve the problems of feature loss and lack of correlation of long-distance information caused by grid effect and other factors in traditional void space pooling;

[0043] (3) The class imbalance function proposed in this invention is beneficial to reducing the impact of the imbalance between the foreground class and the background class on training.

[0044] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0045] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0046] Figure 1 This is a flowchart of a transparent object segmentation method assisted by user interaction information in an embodiment of the present invention;

[0047] Figure 2 This is a structural diagram of the transparent object segmentation method assisted by user interaction information in an embodiment of the present invention;

[0048] Figure 3 This is a diagram of the pooling structure of the aggregated void space in an embodiment of the present invention;

[0049] Figure 4 This is a comparison diagram of the transparent object segmentation method assisted by user interaction information in this embodiment of the invention with other transparent object segmentation methods. Detailed Implementation

[0050] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprising of," etc., will be understood to include the stated elements or components, and does not exclude other elements or other components.

[0052] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0053] Furthermore, to better illustrate the present invention, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that the present invention can be practiced even without certain specific details. In some instances, methods, means, and elements well-known to those skilled in the art have not been described in detail, in order to highlight the spirit of the present invention.

[0054] According to an embodiment of the present invention, a method for segmenting transparent objects with the assistance of user interaction information is provided, the flowchart of which is shown below. Figure 1 As shown, the specific steps include the following:

[0055] Step 101: Convert the auxiliary information generated by the user's multi-point interaction with transparent objects in the image to be segmented into a Gaussian heatmap through Gaussian transformation;

[0056] In this embodiment, the multi-point interaction includes four extreme points (leftmost, rightmost, topmost, and bottommost) of the object to be segmented, additional positive points, and negative points. The extreme points are used to mark the pixel region where the target to be segmented is located, the additional positive points indicate the foreground, and the negative points indicate the background. The Gaussian heatmap generated by the Gaussian transform is the same size as the original image, ensuring that its heatmap information is consistent with the original image. Figure 1 One-to-one correspondence; the four extreme points and positive points are on the same heatmap, and they act on the foreground class. The four extreme points provide spatial information of the object to be segmented to the segmentation network, while the positive points provide feature information that is difficult to capture due to reflection, high transparency, blurred boundaries, etc.

[0057] The negative points are on a separate heatmap, which act on the background class and provide the segmentation network with feature information that is difficult to identify as background inside the target;

[0058] The Gaussian transform uses the Gaussian distribution function (normal distribution function), and its calculation formula is as follows:

[0059]

[0060] Where x and y represent the coordinates of the pixel, x0 and y0 represent the coordinates of the center point, and represents the standard deviation, which is used to measure the degree of deviation of the pixel from the coordinates of the center point.

[0061] Step 102: Combine the converted Gaussian heatmap with the original image to form a 5-channel input;

[0062] In this embodiment, the 5-channel input is generated by converting the user-provided positive point (for indicating the foreground) and negative point (for indicating the background) interaction information into two heatmaps corresponding to the foreground and background respectively after Gaussian transform encoding. These heatmaps are then stitched together with the R, G, and B channels of the original transparent object image.

[0063] Step 103: Extract initial features through the backbone network, and then further refine the feature map by combining multi-scale information and contextual information through aggregated dilated spatial pooling;

[0064] In this embodiment, ResNet50 is used as the backbone network, and aggregated dilated spatial pooling is used to further refine the initial features. The process is as follows: after the input image is processed by the backbone network to extract the initial features, the network is divided into two branches: boundary flow and normal flow. The boundary flow generates boundary information through aggregated dilated spatial pooling and the decoder, while the normal flow, after aggregated dilated spatial pooling, uses the boundary information generated by the boundary flow as a guide in the boundary attention module to generate a refined feature map.

[0065] The aggregated void space pooling further refines the initial features by combining multi-scale information and contextual information, and its process is as follows:

[0066] Image features processed by pooling layers and dilated convolutions with different dilation rates are upsampled and then added to the original features processed by 1×1 convolution. The added features are then subjected to 3×3 convolution for the first feature aggregation. The aggregated features are then concatenated with the original features. The concatenated feature map is then processed by 1×1 convolution and added to the original features. Finally, the added features are subjected to another 1×1 convolution for the second feature fusion, resulting in aggregated multi-scale features.

[0067] Step 104: Balance the foreground and background classes during training using class imbalance loss;

[0068] In this embodiment, class imbalance loss is used to balance the foreground and background classes, and its calculation method is as follows:

[0069]

[0070] l v =O*(G-O')-log(1+e (O-2(o*o')) )

[0071] Where N, N b N f These represent the total number of pixels in the label, the number of background pixels, and the number of foreground pixels, respectively. O represents the predicted image output by the method, O' represents the image after binarizing O (pixels with pixel values ​​greater than or equal to 0 in O have their corresponding pixel values ​​in O' set to 1, otherwise set to 0), and G represents the label.

[0072] Step 105: Separate the foreground and background of the transparent object image based on the refined and balanced feature map, and output the final segmented image;

[0073] In this embodiment, for comprehensive evaluation, the experiment uses Mean Intersection over Union (mIoU) and Pixel Accuracy (Acc), evaluation metrics widely used in semantic segmentation, instance segmentation, and panoptic segmentation. mIoU refers to the average IoU across all categories. For each category, the number of pixels in the segmentation result and the ground truth are calculated, and their intersection and union are calculated. The ratio of the number of pixels in the intersection to the number of pixels in the union is the IoU for that category. Finally, the IoU of all categories is averaged to obtain the average IoU of the entire image—mIoU. This metric can be used to evaluate the performance differences of the method across different categories; a higher value indicates better segmentation performance across different categories. Acc refers to the average pixel classification accuracy of the method across all categories. Specifically, for each category, the number of pixels in the segmentation result and the ground truth are calculated, then the number of correctly classified pixels is calculated, and finally, the pixel classification accuracy of all categories is averaged. Acc can be used to evaluate the overall classification accuracy of the method; a higher value indicates better classification performance. Their calculation formulas are as follows:

[0074]

[0075]

[0076] Where k represents the total number of pixels in the predicted image, i represents the actual pixels, j represents the predicted pixels, and p ij This represents the number of pixels that predict i as j.

[0077] Table 1 shows the performance comparison results of the proposed method with current mainstream transparent object segmentation methods on the Trans10k transparent object segmentation dataset. In the table, bold text indicates the best results, and Easy, Hard, and All represent the performance on the Easy subset, Hard subset, and full test set of the Trans10k transparent object segmentation dataset, respectively. Images in the Easy subset mostly contain only one object and have relatively regular edges or low transparency; images in the Hard subset typically contain multiple objects, with more occlusions and interference, higher transparency, and are more difficult to segment.

[0078] Table 1

[0079]

[0080] To more intuitively demonstrate the segmentation effect of the proposed method on transparent objects, segmentation methods for partially transparent objects of the same type were randomly selected and tested on the Trans10K transparent object segmentation dataset, and the segmentation results were visualized. Figure 4 As shown in the figure. The test images are randomly selected. Each row in the figure represents the segmentation effect of different methods on the same image, and each column represents the segmentation effect of the same method on different images.

[0081] This invention provides a transparent object segmentation method assisted by user interaction information. By using a small amount of user interaction as auxiliary information, it effectively improves the segmentation results of transparent objects. This method can provide high-precision masks for transparent object segmentation tasks and can also serve as an accurate and efficient mask annotation tool, improving the annotation efficiency of datasets.

[0082] The above describes in detail a transparent object segmentation method assisted by user interaction information. Based on this method, a transparent object segmentation device assisted by user interaction information is proposed. The structure and deployment process of the device are described in detail below.

[0083] This invention also provides a transparent object segmentation device assisted by user interaction information. The device consists of four modules, including:

[0084] The user interaction module generates auxiliary information by allowing users to interact with transparent objects in the image to be segmented at multiple points, and outputs an image with the user interaction information added.

[0085] Clicking the encoding module converts the user's multi-point interaction information into a Gaussian heatmap through Gaussian transformation, and combines it with the original image to generate a 5-channel input;

[0086] The feature extraction module performs preliminary feature extraction on the input through the backbone network, and then further refines the features through aggregated dilated spatial pooling.

[0087] The image segmentation module separates the foreground and background of the transparent object image based on the refined and balanced feature map, and outputs the final segmented image.

[0088] In one possible implementation, the multi-point interaction in the user interaction module includes:

[0089] The four extreme points (leftmost, rightmost, topmost, and bottommost) of the object to be segmented are used to provide spatial information of the object to be segmented for the segmentation network.

[0090] Positive points are used to indicate the foreground and provide segmentation networks with feature information that is difficult to capture due to factors such as reflection, high transparency, and blurred boundaries.

[0091] Negative points are used to indicate the background and provide segmentation networks with feature information about the interior of the target that is difficult to identify as background.

[0092] In one possible implementation, the Gaussian transform in the click encoding module uses a Gaussian distribution function (normal distribution function). The 5-channel input consists of user-provided extreme points, positive points (foreground indication), and negative points (for background indication) interactive information. After being encoded by Gaussian transform, these are converted into two heatmaps corresponding to the foreground and background, respectively, which are then spliced ​​together with the R, G, and B channels of the original transparent object image.

[0093] In one possible implementation, the feature extraction module uses ResNet50 as the backbone network and employs aggregated dilated spatial pooling to further refine the initial features. The process is as follows: after the input image is processed by the backbone network to obtain initial features, the network is divided into two branches: a boundary flow and a regular flow. The boundary flow generates boundary information through aggregated dilated spatial pooling and a decoder, while the regular flow, after aggregated dilated spatial pooling, uses the boundary information generated by the boundary flow as a guide in the boundary attention module to generate a refined feature map.

[0094] The aggregated void space pooling further refines the initial features by combining multi-scale information and contextual information, and its process is as follows:

[0095] Image features processed by pooling layers and dilated convolutions with different dilation rates are upsampled and then added to the original features processed by 1×1 convolution. The added features are then subjected to 3×3 convolution for the first feature aggregation. The aggregated features are then concatenated with the original features. The concatenated feature map is then processed by 1×1 convolution and added to the original features. Finally, the added features are subjected to another 1×1 convolution for the second feature fusion, resulting in aggregated multi-scale features.

[0096] The above four modules are encapsulated and deployed on a server platform to obtain a transparent object segmentation device assisted by user interaction information, which is used to achieve transparent object segmentation.

[0097] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0099] The foregoing description of specific exemplary embodiments of the present invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and corresponding changes in form and detail may be made based on the steps described above, further enabling those skilled in the art to implement various embodiments or make various changes and selections using the invention. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A method for transparent object segmentation with user interaction information assistance, characterized in that, Includes the following steps: Step S1: Convert the auxiliary information generated by the user's multi-point interaction with transparent objects in the image to be segmented into a Gaussian heatmap through Gaussian transformation; The multi-point interaction includes four extreme points of the object to be segmented: the leftmost point, the rightmost point, the topmost point, and the bottommost point, as well as additional positive and negative points. The extreme points are used to mark the pixel region where the target is located, the additional positive points indicate the foreground, and the negative points indicate the background. The Gaussian heatmap generated by the Gaussian transform is the same size as the original image, ensuring a one-to-one correspondence between its heatmap information and the original image. The four extreme points and positive points are on the same heatmap, acting on the foreground class. The four extreme points provide spatial information of the object to be segmented to the segmentation network, while the positive points provide the segmentation network with feature information that is difficult to capture due to reflection, high transparency, and blurred boundaries. The negative points are on a separate heatmap, acting on the background class, providing the segmentation network with feature information of the target's interior that is difficult to identify as background. Step S2: Combine the converted Gaussian heatmap with the original image to form a 5-channel input; Step S3: Extract initial features through the backbone network, and then further refine the feature map by combining multi-scale information and contextual information through aggregated spatial pooling; The backbone network uses ResNet50 and employs aggregated dilated spatial pooling to further refine the initial features. The process is as follows: after the input image is processed by the backbone network to extract initial features, the network is divided into two branches: a boundary flow and a regular flow. The boundary flow generates boundary information through aggregated dilated spatial pooling and a decoder, while the regular flow, after aggregated dilated spatial pooling, uses the boundary information generated by the boundary flow as a guide in the boundary attention module to generate a refined feature map. The aggregated void space pooling further refines the initial features by combining multi-scale information and contextual information, and its process is as follows: Image features processed by pooling layers and dilated convolutions with different dilation rates are upsampled and then added to the original features processed by 1×1 convolution. The added features are then subjected to 3×3 convolution for the first feature aggregation. The aggregated features are then concatenated with the original features. The concatenated feature map is then processed by 1×1 convolution and added to the original features. Finally, the added features are subjected to another 1×1 convolution for the second feature fusion, resulting in aggregated multi-scale features. Step S4: Balance the foreground and background classes during training using class imbalance loss; Step S5: Separate the foreground and background of the transparent object image based on the refined and balanced feature map, and output the final segmented image.

2. The transparent object segmentation method assisted by user interaction information according to claim 1, characterized in that, The five channel inputs in step S2 consist of interactive information provided by the user, consisting of positive points indicating the foreground and negative points indicating the background. After being encoded by Gaussian transform, these are converted into two heatmaps corresponding to the foreground and background, respectively, which are then stitched together with the R, G, and B channels of the original image of the transparent object.

3. The transparent object segmentation method assisted by user interaction information according to claim 1, characterized in that, Step S4 uses class-imbalanced loss to balance the foreground and background classes, and its calculation method is as follows: Wherein, N, N b , N f respectively represent the total number of pixels in the label, the number of background pixels, the number of foreground pixels; O represents the predicted image output by the method; O' represents the image after the binary processing of O: the pixel point in O whose pixel value is greater than or equal to 0 is set to 1 in O', otherwise it is set to 0; G represents the label.

4. A transparent object segmentation device assisted by user interaction information, characterized in that, The device consists of four modules, including: The user interaction module generates auxiliary information by allowing users to interact with transparent objects in the image to be segmented at multiple points, and outputs an image with the user interaction information added. Clicking the encoding module converts the user's multi-point interaction information into a Gaussian heatmap through Gaussian transformation, and combines it with the original image to generate a 5-channel input. The feature extraction module performs preliminary feature extraction on the input through the backbone network, and then further refines the features through aggregated dilated spatial pooling. The image segmentation module separates the foreground and background of the transparent object image based on the refined and balanced feature map, and outputs the final segmented image. The multi-point interaction in the user interaction module includes: The four extreme points of the object to be segmented—the leftmost point, the rightmost point, the topmost point, and the bottommost point—are used to provide spatial information of the object to be segmented to the segmentation network. Positive points are used to indicate the foreground and provide segmentation networks with feature information that is difficult to capture due to reflection, high transparency, and blurred boundaries; Negative points are used to indicate the background and to provide segmentation networks with feature information about the interior of the target that is difficult to identify as background. In the click encoding module, the Gaussian transform uses the Gaussian distribution function. The five channel inputs consist of interactive information provided by the user, including extreme points, positive points indicating the foreground, and negative points indicating the background. After being encoded by the Gaussian transform, these are converted into two heatmaps corresponding to the foreground and background, which are then spliced ​​together with the R, G, and B channels of the original transparent object image. The feature extraction module uses ResNet50 as the backbone network and employs aggregated dilated spatial pooling to further refine the initial features. The process is as follows: after the input image is processed by the backbone network to obtain the initial features, the network is divided into two branches: boundary flow and normal flow. The boundary flow generates boundary information through aggregated dilated spatial pooling and the decoder, while the normal flow, after aggregated dilated spatial pooling, uses the boundary information generated by the boundary flow as a guide in the boundary attention module to generate a refined feature map. The aggregated void space pooling further refines the initial features by combining multi-scale information and contextual information, and its process is as follows: Image features processed by pooling layers and dilated convolutions with different dilation rates are upsampled and then added to the original features processed by 1×1 convolution. The added features are then subjected to 3×3 convolution for the first feature aggregation. The aggregated features are then concatenated with the original features. The concatenated feature map is then processed by 1×1 convolution and added to the original features. Finally, the added features are subjected to another 1×1 convolution for the second feature fusion, resulting in aggregated multi-scale features. The above four modules are encapsulated and deployed on a server platform to obtain a transparent object segmentation device assisted by user interaction information, which is used to achieve transparent object segmentation.

Citation Information

Patent Citations

  • Picture processing method and device, electronic equipment and storage medium

    CN116958184A

  • Foreground-background-aware atrous multiscale network for disparity estimation

    US20200160533A1