Image segmentation method, device, equipment and storage medium

By using diffusion model in polar coordinate system to add noise and denoising images, combined with U-NET, the problem of low interactive image segmentation accuracy in the prior art is solved, and higher segmentation accuracy and Feature representation ability.

CN117036696BActive Publication Date: 2024-11-12TSINGHUA SHENZHEN INTERNATIONAL GRADUATE SCHOOL
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310911823.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-07-21
Publication Date
2024-11-12
Estimated Expiration
2043-07-21

AI Technical Summary

Technical Problem

In the prior art, the interactive image segmentation algorithm has poor segmentation accuracy and cannot fully utilize the interactive operation information.

Method used

Using the forward and reverse processes of the diffusion model, the binary image is added and denoised in the polar coordinate system, using interactive operation information as control parameters, combined with the noise reduction function constructed by U-NET in U-NET, gradually improves the Image segmentation accuracy.

Benefits of technology

Through the combination of diffusion model and interactive segmentation algorithm, the segmentation accuracy of interactive image segmentation is improved, interactive operation information is fully utilized, and feature representation and detail recovery capabilities are enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117036696B_ABST
    Figure CN117036696B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of image processing technology, and provides an image segmentation method, device, equipment and storage medium, the method comprising: obtaining a first binary image corresponding to an image to be segmented; using the forward process of a diffusion model, according to interactive operation information, noise obeying a standard Gaussian distribution, and the first binary image, obtaining a second binary image after adding noise, wherein the first binary image and the second binary image are both binary images annotated in a polar coordinate system, and the coordinate origin of the polar coordinate system is determined according to at least one forward annotation point corresponding to the interactive operation information; using the reverse process of the diffusion model, according to the second binary image after adding noise and the image to be segmented, obtaining a first target image. In the present invention, the diffusion model is combined with an interactive segmentation algorithm, the interactive operation information is more fully utilized, and the segmentation accuracy of the interactive image segmentation is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image segmentation method, device, equipment and storage medium. Background Art

[0002] Image segmentation is an important part of artificial intelligence, computer vision and other fields. Interactive image segmentation is achieved by introducing a small number of user markers (such as clicks, strokes, bounding boxes, graffiti and other interactive operations) and using these markers as priors to guide the network to segment the area of ​​interest to the user.

[0003] In the prior art, interactive image segmentation algorithms are mostly based on traditional convolutional neural networks (CNN) or transformer structures, which input interactive operation information together with the image to be segmented into the network to achieve interactive segmentation. However, because the interactive operation information is relatively sparse, the segmentation accuracy of the existing methods is poor. Summary of the invention

[0004] The present invention provides an image segmentation method, device, equipment and storage medium, which are used to solve the defect of poor segmentation accuracy in the prior art and improve the segmentation accuracy of image segmentation.

[0005] In a first aspect, the present invention provides an image segmentation method, the method comprising:

[0006] Obtaining a first binary image corresponding to the image to be segmented;

[0007] Using the forward process of the diffusion model, according to the interactive operation information, the noise obeying the standard Gaussian distribution, and the first binary image, a second binary image after adding noise is obtained; the first binary image and the second binary image are both binary images annotated in a polar coordinate system, and the coordinate origin of the polar coordinate system is determined according to at least one forward annotated point corresponding to the interactive operation information;

[0008] The first target image is obtained according to the second binary image after adding noise and the image to be segmented by utilizing the inverse process of the diffusion model.

[0009] Optionally, the forward process of using the diffusion model to obtain the second binary image after adding noise according to the interactive operation information, the noise obeying the standard Gaussian distribution, and the first binary image includes:

[0010] According to the interactive operation information, a control parameter related to the diffusion amplitude is obtained;

[0011] The second binary image after adding noise is obtained according to the control parameter, the first binary image, and the noise obeying the standard Gaussian distribution.

[0012] Optionally, obtaining the second binary image after adding noise according to the control parameter, the first binary image, and the noise obeying the standard Gaussian distribution includes:

[0013] The following formula (1) and formula (2) are used to obtain the second binary image after adding noise corresponding to each time step according to the control parameter, the first binary image, and the noise obeying the standard Gaussian distribution:

[0014]

[0015]

[0016] Among them, q(x t |x0) represents the second binary image after adding noise corresponding to the t-th time step, x0 represents the first binary image, I represents the noise that obeys the standard Gaussian distribution, β t represents the control parameter corresponding to the tth time step;

[0017] The second binary image after adding noise is obtained according to the second binary image after adding noise corresponding to each of the time steps and the number of time steps corresponding to the diffusion model.

[0018] Optionally, the diffusion model includes a denoising function constructed based on a U-net network, and the inverse process of the diffusion model is used to obtain the first target image according to the second binary image after adding noise and the image to be segmented, including:

[0019] Using the denoising function, taking the second binary image after adding noise as a condition, the image to be segmented is gradually denoised to obtain denoised images corresponding to each time step; the denoised image corresponding to the t-th time step is the image to be segmented input to the t-1-th time step, wherein t is an integer greater than 0 and less than T, and T is the number of time steps corresponding to the diffusion model;

[0020] A first target image is obtained according to the denoised images corresponding to the respective time steps and the number of the time steps.

[0021] Optionally, before obtaining the second binary image after adding noise according to the interactive operation information, the noise obeying the standard Gaussian distribution, and the first binary image by using the forward process of the diffusion model, the method further includes:

[0022] The position information of each pixel point included in the first binary image in the rectangular coordinate system is converted into the position information of each pixel point included in the first binary image in the polar coordinate system.

[0023] Optionally, the interactive operation information is represented by a heat map, and the heat map includes a positive heat map corresponding to a positive click marking the foreground and a negative heat map corresponding to a negative click marking the background.

[0024] In a second aspect, the present invention provides an image segmentation device, the device comprising:

[0025] An acquisition module, used for acquiring a first binary image corresponding to the image to be segmented;

[0026] A processing module, configured to obtain a second binary image after adding noise according to the interactive operation information, the noise obeying the standard Gaussian distribution, and the first binary image by using the forward process of the diffusion model; the first binary image and the second binary image are both binary images annotated in a polar coordinate system, and the coordinate origin of the polar coordinate system is determined according to at least one forward annotated point corresponding to the interactive operation information;

[0027] The first target image is obtained according to the second binary image after adding noise and the image to be segmented by utilizing the inverse process of the diffusion model.

[0028] In a third aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the image segmentation method described above is implemented.

[0029] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described image segmentation methods.

[0030] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image segmentation method described above is implemented.

[0031] The present invention provides an image segmentation method, device, equipment and storage medium. The method obtains a first binary image corresponding to an image to be segmented, and then uses a forward process of a diffusion model to obtain a second binary image after adding noise according to interactive operation information, noise obeying a standard Gaussian distribution, and the first binary image, wherein the first binary image and the second binary image are both binary images annotated in a polar coordinate system, and the coordinate origin of the polar coordinate system is determined according to at least one forward annotated point corresponding to the interactive operation information; and then uses a reverse process of the diffusion model to obtain a first target image according to the second binary image after adding noise and the image to be segmented. In the present invention, based on interactive operation information, the forward process of the diffusion model is used to gradually add noise to the first binary image corresponding to the image to be segmented, and the first binary image is annotated in a polar coordinate system. The diffusion process of adding noise based on the polar coordinate system is more natural, and the accuracy of simulating the binary image is higher; then, the reverse process of the diffusion model is used to obtain the first target image according to the second binary image after adding noise and the image to be segmented. Because the diffusion model itself has good feature representation ability and detail recovery ability, the diffusion model is combined with the interactive segmentation algorithm, the interactive operation information is more fully utilized, and the segmentation accuracy of the interactive image segmentation is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 It is one of the flow charts of the image segmentation method provided by the present invention;

[0034] Figure 2 This is the second flow chart of the image segmentation method provided by the present invention;

[0035] Figure 3 It is a structural schematic diagram of the image segmentation device provided by the present invention;

[0036] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first" and "second" are generally a class, and the number of objects is not limited, for example, the first node can be one or more. "And / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the front and back associated objects are a kind of "or" relationship. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or equipment.

[0039] In order to more clearly understand the embodiments provided by the present invention, relevant technical terms are first introduced.

[0040] Interactive segmentation: refers to the task of segmenting the target foreground and background through a small number of interactive operations such as clicking and drawing by the user. Compared with general semantic segmentation, interactive segmentation is often more targeted and has the characteristics of higher segmentation accuracy for single objects.

[0041] Heatmap: Also known as a heat map, it is a diagram that shows the user's page click location or the user's page location in a special highlighted form. With the help of a heatmap, you can intuitively observe the user's overall access and click preferences.

[0042] Binary Image: It means that each pixel on the image has only two possible values ​​or grayscale levels. People often use black and white, B&W, and monochrome images to represent binary images.

[0043] Combine the following Figure 1-Figure 4 The image segmentation solution provided by the present invention is described.

[0044] Figure 1It is one of the flowcharts of the image segmentation method provided by the present invention, such as Figure 1 As shown, the method includes:

[0045] Step 101, obtaining a first binary image corresponding to the image to be segmented;

[0046] Specifically, it can be understood that a binary image refers to an image with only two grayscale levels, that is, the grayscale value of any pixel in the image is 0 or 255, representing black and white respectively. The main applications of obtaining binary images are such as edge extraction and information hiding. In this embodiment, binary images are mainly used to extract the edges of target objects.

[0047] In this step, it is necessary to first perform a binarization operation on the image to be segmented, that is, to set the grayscale value of each pixel on the image to be segmented to 0 or 255, that is, to make the entire image to be segmented present an obvious visual effect of only black and white, and the image to be segmented after the binarization operation is determined as the first binary image.

[0048] Step 102: using the forward process of the diffusion model, according to the interactive operation information, the noise obeying the standard Gaussian distribution, and the first binary image, a second binary image after adding noise is obtained; the first binary image and the second binary image are both binary images annotated in a polar coordinate system, and the coordinate origin of the polar coordinate system is determined according to at least one positive annotated point corresponding to the interactive operation information;

[0049] Specifically, after obtaining the first binary image, in step 102, based on the first binary image, the forward process of the diffusion model can be used, combined with the interactive operation information and the noise obeying the standard Gaussian distribution, to gradually add noise to the first binary image, wherein the interactive operation information is represented, for example, in the form of a heat map, and the first binary image is a binary image annotated in a polar coordinate system, and the origin of the polar coordinate system is the forward annotation point.

[0050] It can be understood that, when gradually adding noise to the first binary image marked in the polar coordinate system, a second binary image after adding noise corresponding to each time step can be obtained. Furthermore, the second binary image after adding noise corresponding to each time step and the number of time steps corresponding to the diffusion model can be used to obtain the second binary image after adding noise. The second binary image after adding noise is, for example, an irregular image similar to a circle.

[0051] Step 103: using the inverse process of the diffusion model, a first target image is obtained according to the second binary image after adding noise and the image to be segmented.

[0052] Specifically, after obtaining the second binary image after adding noise, the inverse process of the diffusion model can be used in combination with the image to be segmented to gradually perform denoising processing to obtain the denoised images corresponding to each time step. Then, the first target image can be obtained according to the denoised images corresponding to each time step and the number of diffusion steps corresponding to the diffusion model, that is, the number of time steps. For example, the denoised image corresponding to the first time step is used as the first target image to complete the interactive segmentation of the image.

[0053] In the method provided in this embodiment, a first binary image corresponding to the image to be segmented is obtained, and then a second binary image after adding noise is obtained by using the forward process of the diffusion model according to the interactive operation information, the noise obeying the standard Gaussian distribution, and the first binary image, wherein the first binary image and the second binary image are both binary images annotated in a polar coordinate system, and the coordinate origin of the polar coordinate system is determined according to at least one forward annotated point corresponding to the interactive operation information; and then, a first target image is obtained by using the reverse process of the diffusion model according to the second binary image after adding noise and the image to be segmented. In the present invention, based on interactive operation information, the forward process of the diffusion model is used to gradually add noise to the first binary image corresponding to the image to be segmented, and the first binary image is annotated in a polar coordinate system. The diffusion process of adding noise based on the polar coordinate system is more natural, and the accuracy of simulating the binary image is higher; then, the reverse process of the diffusion model is used to obtain the first target image according to the second binary image after adding noise and the image to be segmented. Because the diffusion model itself has good feature representation ability and detail recovery ability, the diffusion model is combined with the interactive segmentation algorithm, the interactive operation information is more fully utilized, and the segmentation accuracy of the interactive image segmentation is improved.

[0054] Optionally, using a forward process of a diffusion model, according to the interactive operation information, the noise obeying a standard Gaussian distribution, and the first binary image, a second binary image after adding noise is obtained, including:

[0055] According to the interactive operation information, the control parameters related to the diffusion amplitude are obtained;

[0056] According to the control parameter, the first binary image, and the noise obeying the standard Gaussian distribution, a second binary image after adding noise is obtained.

[0057] Specifically, step 102 can be implemented by the following steps:

[0058] First, based on the interactive operation information, such as the heat map, the control parameters related to the diffusion amplitude are calculated. The process is shown below:

[0059]

[0060] Among them, β t represents the control parameter corresponding to the t-th time step, heat represents the value of the corresponding pixel in the heat map corresponding to the interactive operation information, T represents the number of time steps corresponding to the diffusion model, and t represents the t-th time step;

[0061] Furthermore, the diffusion amplitude of each time step can be controlled according to the control parameter, and the image to be segmented input at each time step and the noise obeying the standard Gaussian distribution can be combined to obtain the second binary image after adding noise corresponding to each time step;

[0062] Furthermore, the final second binary image after adding noise can be obtained based on the second binary image after adding noise corresponding to each time step and the number of time steps corresponding to the diffusion model. For example, if the number of time steps corresponding to the diffusion model is T, the second binary image after adding noise corresponding to the Tth time step is determined as the final second binary image after adding noise.

[0063] In the method provided in this embodiment, by using the forward process of the diffusion model, according to the interactive operation information, a control parameter related to the diffusion amplitude is obtained, and then according to the control parameter, the first binary image, and the noise obeying the standard Gaussian distribution, a second binary image after adding noise is obtained. This method uses the interactive operation information of interactive segmentation to constrain the control parameters of the diffusion model, so that the interactive operation information is maximized, and the full use of the interactive operation information is achieved.

[0064] Optionally, obtaining a second binary image after adding noise according to the control parameter, the first binary image, and noise obeying a standard Gaussian distribution includes:

[0065] The following formula (1) and formula (2) are used to obtain the second binary image after adding noise corresponding to each time step according to the control parameters, the first binary image, and the noise obeying the standard Gaussian distribution:

[0066]

[0067]

[0068] Among them, q(x t |x0) represents the second binary image after adding noise corresponding to the t-th time step, x0 represents the first binary image, I represents the noise that obeys the standard Gaussian distribution, β t represents the control parameter corresponding to the tth time step;

[0069] According to the second binary image after adding noise corresponding to each time step and the number of time steps corresponding to the diffusion model, the second binary image after adding noise is obtained.

[0070] Specifically, it should be noted that β t It is a parameter defined artificially in the diffusion process, which can be used to control the diffusion amplitude. t The larger the value of , the higher the randomness of the diffusion result. t The smaller the value of , the closer the diffusion result is to the diffusion starting point, and the higher the certainty of the diffusion result.

[0071] According to the control parameters, the first binary image, and the noise obeying the standard Gaussian distribution, the second binary image after adding noise corresponding to each time step can be obtained. The process is exemplified as follows:

[0072] Assume that the first binary image is x0 (which can also be understood as the diffusion starting point in the expansion process), and the control parameter is β t , the second binary image after adding noise corresponding to the t-th time step can be calculated using the following formula (1) and formula (2):

[0073]

[0074]

[0075] Among them, q(x t |x0) represents the second binary image after adding noise corresponding to the t-th time step, x0 represents the first binary image (that is, the diffusion starting point in the expansion process), I represents the noise that obeys the standard Gaussian distribution, β t represents the control parameter corresponding to the tth time step; α t is 1-β t The result of the cumulative multiplication is α for the simplicity of the formula. t express;

[0076] After obtaining the second binary image after adding noise corresponding to each time step, the first target image can be obtained based on the second binary image after adding noise corresponding to each time step and the number of time steps corresponding to the diffusion model. The process is exemplified as follows:

[0077] Assuming that the number of time steps corresponding to the diffusion model is T, t is updated from 1 to T, and the above process is iterated to obtain the second binary image after adding noise corresponding to each of the T time steps. The second binary image after adding noise corresponding to the Tth time step can be determined as the second binary image.

[0078] In the method provided in this embodiment, the second binary image after adding noise corresponding to each time step is obtained according to the control parameters, the first binary image, and the noise obeying the standard Gaussian distribution. Then, the second binary image after adding noise corresponding to each time step and the number of time steps corresponding to the diffusion model are used to obtain the second binary image after adding noise, thereby achieving the adaptability of the diffusion model to the interactive segmentation. Based on this, a more accurate image segmentation result can be obtained through the inverse process of the diffusion model.

[0079] Optionally, the diffusion model includes a denoising function constructed based on a U-net network, and the first target image is obtained according to the second binary image after adding noise and the image to be segmented by using the inverse process of the diffusion model, including:

[0080] Using the denoising function, the second binary image after adding noise is used as a condition to gradually denoise the image to be segmented, and obtain the denoised image corresponding to each time step; the denoised image corresponding to the t-th time step is the image to be segmented input to the t-1-th time step, where t is an integer greater than 0 and less than T, and T is the number of time steps corresponding to the diffusion model;

[0081] A first target image is obtained according to the denoised images corresponding to each time step and the number of time steps.

[0082] Specifically, the diffusion model in this implementation includes a denoising function constructed based on a U-net network, and the inverse process of the diffusion model is used, that is, the denoising function constructed based on the U-net network is used to obtain the first target image according to the second binary image after adding noise and the image to be segmented. The specific example of this process is as follows:

[0083] First, using a denoising function constructed based on a U-net network, for example, the second binary image after adding noise is used as a condition to gradually denoise the image to be segmented, and obtain the denoised image corresponding to each time step, wherein the denoised image corresponding to the t-th time step is the image to be segmented input to the t-1-th time step. Assuming that T is the number of time steps corresponding to the diffusion model, and t is an integer greater than 0 and less than T, an example of the process of obtaining the denoised image corresponding to each time step is as follows:

[0084] The image to be segmented corresponding to the T-th time step (that is, the original image to be segmented) and the second binary image after adding noise (that is, the second binary image after adding noise corresponding to the T-th time step) are input into the U-NET corresponding to the T-th time step, and the noise distribution of the second binary image after adding noise is learned by using the U-NET, and the image to be segmented corresponding to the T-th time step is denoised to obtain the denoised image corresponding to the T-th time step;

[0085] Furthermore, the denoised image corresponding to the T-th time step and the noisy image corresponding to the T-1-th time step in the forward process of the diffusion model are input together into the U-NET corresponding to the T-1-th time step, and the U-NET is used to learn the noise distribution of the noisy image corresponding to the T-1-th time step in the forward process of the diffusion model, and the noisy image corresponding to the T-1-th time step is denoised to obtain the denoised image corresponding to the T-1-th time step;

[0086] Secondly, the first target image can be obtained according to the denoised images corresponding to each time step and the number of time steps. The process is exemplified as follows:

[0087] By updating t from T to 1 and iterating the above process, the denoised images corresponding to t time steps can be obtained. The denoised image corresponding to the first time step can be determined as the first target image.

[0088] In the method provided in this embodiment, by using a denoising function, the second binary image after adding noise is used as a condition to gradually denoise the image to be segmented, obtain the denoised image corresponding to each time step, and then obtain the first target image according to the denoised image corresponding to each time step and the number of time steps. The method provided in this embodiment realizes the adaptability of the diffusion model and interactive segmentation, based on which a more accurate image segmentation result can be obtained through the inverse process of the diffusion model.

[0089] Optionally, before obtaining the second binary image after adding noise according to the interaction operation information, the noise obeying the standard Gaussian distribution, and the first binary image by using the forward process of the diffusion model, the method further includes:

[0090] The position information of each pixel point included in the first binary image in the rectangular coordinate system is converted into the position information of each pixel point included in the first binary image in the polar coordinate system.

[0091] Specifically, it is understandable that in the usual segmentation method, the binary image is represented based on the rectangular coordinate system. However, if it is a binary image containing interactive operation information, the use of the plane rectangular coordinate system to represent the contour information of the object is slightly insufficient. Therefore, in this embodiment, the polar coordinate system can be used to represent the binary image.

[0092] Among them, the first binary image based on the rectangular coordinate system can be converted into the first binary image based on the polar coordinate system. Since the image is usually represented pixel by pixel, this step can specifically be to convert the position information of each pixel point contained in the first binary image in the rectangular coordinate system into the position information of each pixel point contained in the first binary image in the polar coordinate system. Among them, the process of mutual conversion between the rectangular coordinate system and the polar coordinate system of each pixel point is as follows:

[0093] x i = cosθ i ×d i +x c (4)

[0094] y i = sinθ i ×d i +y c (5)

[0095] Among them, (x i ,y i ) represents the coordinate of the i-th pixel in the rectangular coordinate system, (d i ,θ i, ) represents the coordinate of the i-th pixel in the polar coordinate system, θ i represents the polar angle of the i-th pixel, d i represents the polar radius of the i-th pixel, (x c ,y c ) represents the coordinate representation of the pole of the polar coordinate system in the rectangular coordinate system, that is, the position of the positive annotation point.

[0096] In the method provided in this embodiment, the binary image is represented by a polar coordinate system. Compared with a plane rectangular coordinate system, polar coordinates have greater advantages in outlining physical contours. By controlling the density of rays, the accuracy of the contour can be freely selected. When the accuracy requirement is not high, the contour of the object can be well represented by the length of only a few rays. In addition, the binary image denoising process based on the polar coordinate system is more natural and can simulate the characteristics of the binary image as much as possible.

[0097] Optionally, the interactive operation information is represented by a heat map, which includes a positive heat map corresponding to a positive click marking the foreground and a negative heat map corresponding to a negative click marking the background.

[0098] Specifically, it is understandable that with the help of heat maps, the user's overall access and click preferences can be intuitively observed. Several common heat maps include heat maps based on mouse click positions, heat maps based on mouse movement trajectories, and heat maps based on content clicks. Among them, the absolute position of the mouse click within a relative time will be recorded in the heat map based on the mouse click position. In this embodiment, the heat map is mainly a click-based heat map, which can be divided into positive heat maps and negative heat maps according to the purpose of the click operation. The positive heat map refers to the heat map that highlights the position corresponding to the positive click that marks the target foreground, and correspondingly, the negative heat map refers to the heat map that highlights the position corresponding to the negative click that marks the background.

[0099] Optionally, the interactive operation information in this embodiment may also have other representation forms, such as the origin of a polar coordinate system, the input of a forward process of a diffusion model, etc.

[0100] In the method provided in this embodiment, the interactive operation information such as heat map can intuitively reflect the user's segmentation intention, and also facilitate the subsequent diffusion model to better perform image segmentation based on the interactive operation information.

[0101] Figure 2 FIG. 2 is a flow chart of the image segmentation method provided by the present invention. Figure 2 As shown, it is a schematic diagram of the process of training a diffusion model, and the method includes:

[0102] First, the sample image can be binarized to obtain a binary image corresponding to the sample image;

[0103] Then, the diffusion model is trained based on the sample image and the binary image corresponding to the sample image. The training steps are as follows:

[0104] Specifically, the forward process of the diffusion model is used to gradually add noise to the binary image corresponding to the sample image according to the heat maps corresponding to the interactive operation information, such as the positive clicks of the marked target and the negative clicks of the marked background, to obtain the binary image after adding noise corresponding to the sample image; then, the binary image after adding noise corresponding to the sample image and the sample image are input into the diffusion model together, and the reverse process of the diffusion model is used to learn the noise distribution in the binary image after adding noise corresponding to the sample image, and the sample image is gradually denoised to obtain the second target image;

[0105] Optionally, the diffusion model may be evaluated based on a model evaluation index, such as an image similarity index, and the model parameters may be adjusted based on the evaluation result. The trained diffusion model may be obtained according to the adjusted model parameters.

[0106] The image segmentation device 300 provided by the present invention is described below. The image segmentation device described below and the image segmentation method described above can be referred to each other.

[0107] Figure 3 3 is a schematic diagram of the structure of the image segmentation device 300 provided by the present invention. Figure 3 As shown, the device comprises:

[0108] An acquisition module 310 is used to acquire a first binary image corresponding to the image to be segmented;

[0109] The processing module 320 is used to obtain a second binary image after adding noise according to the interactive operation information, the noise obeying the standard Gaussian distribution, and the first binary image by using the forward process of the diffusion model; the first binary image and the second binary image are both binary images annotated in a polar coordinate system, and the coordinate origin of the polar coordinate system is determined according to at least one forward annotated point corresponding to the interactive operation information;

[0110] The first target image is obtained according to the second binary image after adding noise and the image to be segmented by utilizing the inverse process of the diffusion model.

[0111] In the device provided by the present embodiment, a first binary image corresponding to the image to be segmented is obtained, and then a second binary image after adding noise is obtained according to the interactive operation information, noise obeying the standard Gaussian distribution, and the first binary image by using the forward process of the diffusion model, wherein the first binary image and the second binary image are both binary images annotated in a polar coordinate system, and the coordinate origin of the polar coordinate system is determined according to at least one forward annotated point corresponding to the interactive operation information; and then, a first target image is obtained according to the second binary image after adding noise and the image to be segmented by using the reverse process of the diffusion model. In the present invention, based on interactive operation information, the forward process of the diffusion model is used to gradually add noise to the first binary image corresponding to the image to be segmented, and the first binary image is annotated in a polar coordinate system. The diffusion process of adding noise based on the polar coordinate system is more natural, and the accuracy of simulating the binary image is higher; then, the reverse process of the diffusion model is used to obtain the first target image according to the second binary image after adding noise and the image to be segmented. Because the diffusion model itself has good feature representation ability and detail recovery ability, the diffusion model is combined with the interactive segmentation algorithm, the interactive operation information is more fully utilized, and the segmentation accuracy of the interactive image segmentation is improved.

[0112] Optionally, the processing module 320 is specifically configured to:

[0113] According to the interactive operation information, a control parameter related to the diffusion amplitude is obtained;

[0114] The second binary image after adding noise is obtained according to the control parameter, the first binary image, and the noise obeying the standard Gaussian distribution.

[0115] Optionally, the processing module 320 is further configured to:

[0116] The following formula (1) and formula (2) are used to obtain the second binary image after adding noise corresponding to each time step according to the control parameter, the first binary image, and the noise obeying the standard Gaussian distribution:

[0117]

[0118]

[0119] Among them, q(x t |x0) represents the second binary image after adding noise corresponding to the t-th time step, x0 represents the first binary image, I represents the noise that obeys the standard Gaussian distribution, β t represents the control parameter corresponding to the tth time step;

[0120] The second binary image after adding noise is obtained according to the second binary image after adding noise corresponding to each of the time steps and the number of time steps corresponding to the diffusion model.

[0121] Optionally, the diffusion model includes a noise reduction function constructed based on a U-net network; the processing module 320 is further used for:

[0122] Using the denoising function, taking the second binary image after adding noise as a condition, the image to be segmented is gradually denoised to obtain denoised images corresponding to each time step; the denoised image corresponding to the t-th time step is the image to be segmented input to the t-1-th time step, wherein t is an integer greater than 0 and less than T, and T is the number of time steps corresponding to the diffusion model;

[0123] A first target image is obtained according to the denoised images corresponding to the respective time steps and the number of the time steps.

[0124] Optionally, the image segmentation device 300 further includes a coordinate conversion module;

[0125] The coordinate conversion module is used for:

[0126] The position information of each pixel point included in the first binary image in the rectangular coordinate system is converted into the position information of each pixel point included in the first binary image in the polar coordinate system.

[0127] Optionally, the interactive operation information is represented by a heat map, and the heat map includes a positive heat map corresponding to a positive click marking the foreground and a negative heat map corresponding to a negative click marking the background.

[0128] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the image segmentation method, which includes:

[0129] Obtaining a first binary image corresponding to the image to be segmented;

[0130] Using the forward process of the diffusion model, according to the interactive operation information, the noise obeying the standard Gaussian distribution, and the first binary image, a second binary image after adding noise is obtained; the first binary image and the second binary image are both binary images annotated in a polar coordinate system, and the coordinate origin of the polar coordinate system is determined according to at least one forward annotated point corresponding to the interactive operation information;

[0131] The first target image is obtained according to the second binary image after adding noise and the image to be segmented by utilizing the inverse process of the diffusion model.

[0132] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0133] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the image segmentation method provided by the above methods, the method includes:

[0134] Obtaining a first binary image corresponding to the image to be segmented;

[0135] Using the forward process of the diffusion model, according to the interactive operation information, the noise obeying the standard Gaussian distribution, and the first binary image, a second binary image after adding noise is obtained; the first binary image and the second binary image are both binary images annotated in a polar coordinate system, and the coordinate origin of the polar coordinate system is determined according to at least one forward annotated point corresponding to the interactive operation information;

[0136] The first target image is obtained according to the second binary image after adding noise and the image to be segmented by utilizing the inverse process of the diffusion model.

[0137] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image segmentation method provided by the above methods is implemented, and the method comprises:

[0138] Obtaining a first binary image corresponding to the image to be segmented;

[0139] Using the forward process of the diffusion model, according to the interactive operation information, the noise obeying the standard Gaussian distribution, and the first binary image, a second binary image after adding noise is obtained; the first binary image and the second binary image are both binary images annotated in a polar coordinate system, and the coordinate origin of the polar coordinate system is determined according to at least one forward annotated point corresponding to the interactive operation information;

[0140] The first target image is obtained according to the second binary image after adding noise and the image to be segmented by utilizing the inverse process of the diffusion model.

[0141] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0142] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image segmentation method, characterized in that: include: Obtaining a first binary image corresponding to the image to be segmented; Converting the position information of each pixel point included in the first binary image in a rectangular coordinate system into the position information of each pixel point included in the first binary image in a polar coordinate system; The converting the position information of each pixel point included in the first binary image in the rectangular coordinate system into the position information of each pixel point included in the first binary image in the polar coordinate system includes: x i =cosθ i ×d i +x c (4) and i =sinθ i ×d i +y c (5) Among them, (x i ,y i ) represents the coordinate of the i-th pixel in the rectangular coordinate system, (d i ,θ i, ) represents the coordinate of the i-th pixel in the polar coordinate system, θ i represents the polar angle of the i-th pixel, d i represents the polar radius of the i-th pixel, (x c ,y c ) represents the coordinate representation of the pole of the polar coordinate system in the rectangular coordinate system, that is, the position of the positive mark point; Using the forward process of the diffusion model, a second binary image after adding noise is obtained according to the interactive operation information, the noise obeying the standard Gaussian distribution, and the first binary image; the first binary image and the second binary image are both binary images annotated in a polar coordinate system, and the coordinate origin of the polar coordinate system is determined according to at least one positive annotated point corresponding to the interactive operation information; the interactive operation information is represented by a heat map, and the heat map includes a positive heat map corresponding to a positive click marking the foreground and a negative heat map corresponding to a negative click marking the background; the second binary image after adding noise is an irregular image similar to a circle; Using the inverse process of the diffusion model, a first target image is obtained according to the second binary image after adding noise and the image to be segmented; The forward process of using the diffusion model to obtain a second binary image after adding noise according to the interactive operation information, the noise obeying the standard Gaussian distribution, and the first binary image includes: According to the interactive operation information, a control parameter related to the diffusion amplitude is obtained; According to the control parameter, the first binary image, and the noise obeying the standard Gaussian distribution, obtaining the second binary image after adding the noise; The obtaining of control parameters related to the diffusion amplitude according to the interactive operation information includes: The control parameters related to the diffusion amplitude are obtained according to the interactive operation information using the following formula (3): Among them, β t represents the control parameter corresponding to the t-th time step, heat represents the value of the corresponding pixel in the heat map corresponding to the interactive operation information, T represents the number of time steps corresponding to the diffusion model, and t represents the t-th time step; The step of obtaining the second binary image after adding noise according to the control parameter, the first binary image, and the noise obeying the standard Gaussian distribution comprises: The following formula (1) and formula (2) are used to obtain the second binary image after adding noise corresponding to each time step according to the control parameter, the first binary image, and the noise obeying the standard Gaussian distribution: Among them, q(x t |x0) represents the second binary image after adding noise corresponding to the t-th time step, x0 represents the first binary image, I represents the noise that obeys the standard Gaussian distribution, β t represents the control parameter corresponding to the tth time step; The second binary image after adding noise is obtained according to the second binary image after adding noise corresponding to each of the time steps and the number of time steps corresponding to the diffusion model.

2. The image segmentation method according to claim 1, characterized in that: The diffusion model includes a denoising function constructed based on a U-net network, and the inverse process of the diffusion model is used to obtain a first target image according to the second binary image after adding noise and the image to be segmented, including: Using the denoising function, taking the second binary image after adding noise as a condition, the image to be segmented is gradually denoised to obtain denoised images corresponding to each time step; the denoised image corresponding to the t-th time step is the image to be segmented input to the t-1-th time step, wherein t is an integer greater than 0 and less than T, and T is the number of time steps corresponding to the diffusion model; A first target image is obtained according to the denoised images corresponding to the respective time steps and the number of the time steps.

3. An image segmentation device, characterized in that: include: An acquisition module, used for acquiring a first binary image corresponding to the image to be segmented; A coordinate conversion module, used for converting the position information of each pixel point contained in the first binary image in a rectangular coordinate system into the position information of each pixel point contained in the first binary image in a polar coordinate system; The converting the position information of each pixel point included in the first binary image in the rectangular coordinate system into the position information of each pixel point included in the first binary image in the polar coordinate system includes: x i =cosθ i ×d i +x c (4) and i =sinθ i ×d i +y c (5) Among them, (x i ,y i ) represents the coordinate of the i-th pixel in the rectangular coordinate system, (d i ,θ i, ) represents the coordinate of the i-th pixel in the polar coordinate system, θ i represents the polar angle of the i-th pixel, d i represents the polar radius of the i-th pixel, (x c ,y c ) represents the coordinate representation of the pole of the polar coordinate system in the rectangular coordinate system, that is, the position of the positive mark point; A processing module, for obtaining a second binary image after adding noise by using the forward process of a diffusion model according to interactive operation information, noise obeying a standard Gaussian distribution, and the first binary image; the first binary image and the second binary image are both binary images annotated in a polar coordinate system, and the coordinate origin of the polar coordinate system is determined according to at least one positive annotated point corresponding to the interactive operation information; the interactive operation information is represented by a heat map, and the heat map includes a positive heat map corresponding to a positive click marking a foreground and a negative heat map corresponding to a negative click marking a background; the second binary image after adding noise is an irregular image similar to a circle; The first target image is obtained according to the second binary image after adding noise and the image to be segmented by using the reverse process of the diffusion model; the second binary image after adding noise is obtained according to the interactive operation information, the noise obeying the standard Gaussian distribution, and the first binary image by using the forward process of the diffusion model, including: obtaining control parameters related to the diffusion amplitude according to the interactive operation information; obtaining the second binary image after adding noise according to the control parameters, the first binary image, and the noise obeying the standard Gaussian distribution; the obtaining control parameters related to the diffusion amplitude according to the interactive operation information includes: The control parameters related to the diffusion amplitude are obtained according to the interactive operation information using the following formula (3): Among them, β t represents the control parameter corresponding to the t-th time step, heat represents the value of the corresponding pixel in the heat map corresponding to the interactive operation information, T represents the number of time steps corresponding to the diffusion model, and t represents the t-th time step; The step of obtaining the second binary image after adding noise according to the control parameter, the first binary image, and the noise obeying the standard Gaussian distribution comprises: obtaining the second binary image after adding noise corresponding to each time step according to the control parameter, the first binary image, and the noise obeying the standard Gaussian distribution using the following formula (1) and formula (2): Among them, q(x t |x0) represents the second binary image after adding noise corresponding to the t-th time step, x0 represents the first binary image, I represents the noise that obeys the standard Gaussian distribution, β t represents the control parameter corresponding to the tth time step; The second binary image after adding noise is obtained according to the second binary image after adding noise corresponding to each of the time steps and the number of time steps corresponding to the diffusion model.

4. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the image segmentation method as claimed in claim 1 or 2 is implemented.

5. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image segmentation method according to claim 1 or 2 is implemented.

6. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image segmentation method according to claim 1 or 2 is implemented.