Image segmentation method and device based on data generation and computer equipment

Abstract

The invention relates to an image segmentation method and device based on data generation, computer equipment and a computer readable storage medium. The method comprises the following steps: acquiring a mask image; the mask image comprises a mask area; inputting the mask image and the preset noise image into a first-stage diffusion model, and performing denoising processing on the preset noise image based on the mask image through the first-stage diffusion model to obtain a low-resolution synthetic image comprising a target object conforming to the mask region in shape; the low-resolution synthetic image is input into a second-stage diffusion model cascaded with the first-stage diffusion model, denoising processing is carried out on the low-resolution synthetic image through the second-stage diffusion model, a high-resolution synthetic image corresponding to the low-resolution synthetic image is obtained, and the high-resolution synthetic image is used for image segmentation model training; the image segmentation model is used for image segmentation. By adopting the method, a compliant model training image can be obtained.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer device and computer-readable storage medium for image segmentation based on data generation. Background Art

[0002] With the help of advanced deep learning and computer vision techniques, deep image segmentation models can obtain valuable semantic information from image datasets. Deep learning models rely on large-scale, diverse, and high-quality training samples for training to obtain high-performance models.

[0003] Currently, collecting large-scale training sample sets is both challenging and costly. In some fields, such as finance and medicine, establishing a large-scale training sample set may face restrictions in terms of privacy, image ownership, etc. For example, the training sample images in the training sample set may be non-compliant due to image ownership issues. Therefore, obtaining compliant training sample images has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] Based on this, it is necessary to provide an image segmentation method, device, computer equipment, computer-readable storage medium and computer program product based on data generation that can be generated and trained by multiple centers in order to solve the above technical problems.

[0005] In a first aspect, the present application provides an image segmentation method based on data generation, comprising:

[0006] Acquire a mask image; the mask image includes a mask area;

[0007] Inputting the mask image and the preset noise image into a first-stage diffusion model, and performing denoising processing on the preset noise image based on the mask image by the first-stage diffusion model to obtain a low-resolution synthetic image including a target object having a region shape consistent with the mask region;

[0008] The low-resolution synthetic image is input into a second-stage diffusion model cascaded with the first-stage diffusion model, and the first low-resolution image is denoised by the second-stage diffusion model to obtain a high-resolution synthetic image corresponding to the low-resolution synthetic image; the high-resolution synthetic image is used for image segmentation model training; and the image segmentation model is used for image segmentation.

[0009] In one embodiment, the denoising process is performed on the preset noise image based on the mask image by using the first stage diffusion model to obtain a low-resolution synthetic image including a target object having a region shape consistent with the mask region, including:

[0010] Determine a target area corresponding to the mask area in the preset noise map;

[0011] Taking the target area as a condition, the preset noise map is subjected to multiple rounds of denoising processing to obtain a low-resolution synthetic image including the target object having a regional shape consistent with the mask area.

[0012] In one embodiment, the step of inputting the low-resolution synthetic image into a second-stage diffusion model cascaded with the first-stage diffusion model, and performing denoising on the low-resolution synthetic image by the second-stage diffusion model to obtain a high-resolution synthetic image corresponding to the low-resolution synthetic image includes:

[0013] Performing upsampling processing on the low-resolution synthetic image to obtain an upsampled image;

[0014] The upsampled image is subjected to multiple rounds of denoising processing through the second-stage diffusion model to obtain a high-resolution synthetic image corresponding to the low-resolution synthetic image.

[0015] In one embodiment, the method further comprises:

[0016] The high-resolution composite image and the mask image are sent to a server, so that the server generates a composite data set based on the high-resolution composite image and the mask image sent by each client, trains the image segmentation model to be trained based on the composite data set, and returns the trained image segmentation model to each client.

[0017] In one embodiment, the training of the image segmentation model to be trained based on the synthetic data set includes:

[0018] For each mask image in the synthetic data set, adjusting the pixel values ​​of the pixel points on the boundary area in the mask image based on a preset smoothing parameter to obtain a mask image with adjusted pixel values;

[0019] The mask image with adjusted pixel values ​​is used as a label to train the image segmentation model to be trained.

[0020] In one embodiment, the method further comprises:

[0021] Sending the high-resolution synthetic image and the mask map to a server, so that the server generates a synthetic data set based on the high-resolution synthetic image and the mask map sent by each client;

[0022] Receiving the synthetic data set returned by the server, and training the image segmentation model to be trained based on the preset local data set and the synthetic data set;

[0023] The image with segmentation is segmented by using the trained image segmentation model to obtain a segmentation result, and the segmentation result is returned to the server, so that the server performs integrated learning based on the segmentation results sent by each of the clients to obtain an integrated result.

[0024] In a second aspect, the present application also provides an image segmentation device based on data generation, comprising:

[0025] An image acquisition module, used to acquire a mask image; the mask image includes a mask area;

[0026] a low-resolution image generation module, configured to input the mask image and the preset noise image into a first-stage diffusion model, and perform denoising processing on the preset noise image based on the mask image through the first-stage diffusion model to obtain a low-resolution synthetic image including a target object having a region shape consistent with the mask region;

[0027] A high-resolution image generation module is used to input the low-resolution synthetic image into a second-stage diffusion model cascaded with the first-stage diffusion model, and perform denoising on the low-resolution synthetic image through the second-stage diffusion model to obtain a high-resolution synthetic image corresponding to the low-resolution synthetic image; the high-resolution synthetic image is used to train an image segmentation model; and the image segmentation model is used to perform image segmentation.

[0028] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0029] Acquire a mask image; the mask image includes a mask area;

[0030] Inputting the mask image and the preset noise image into a first-stage diffusion model, and performing denoising processing on the preset noise image based on the mask image by the first-stage diffusion model to obtain a low-resolution synthetic image including a target object having a region shape consistent with the mask region;

[0031] The low-resolution synthetic image is input into a second-stage diffusion model cascaded with the first-stage diffusion model, and the low-resolution synthetic image is denoised by the second-stage diffusion model to obtain a high-resolution synthetic image corresponding to the low-resolution synthetic image; the high-resolution synthetic image is used for image segmentation model training; and the image segmentation model is used for image segmentation.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0033] Acquire a mask image; the mask image includes a mask area;

[0034] Inputting the mask image and the preset noise image into a first-stage diffusion model, and performing denoising processing on the preset noise image based on the mask image by the first-stage diffusion model to obtain a low-resolution synthetic image including a target object having a region shape consistent with the mask region;

[0035] The low-resolution synthetic image is input into a second-stage diffusion model cascaded with the first-stage diffusion model, and the low-resolution synthetic image is denoised by the second-stage diffusion model to obtain a high-resolution synthetic image corresponding to the low-resolution synthetic image; the high-resolution synthetic image is used for image segmentation model training; and the image segmentation model is used for image segmentation.

[0036] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0037] Acquire a mask image; the mask image includes a mask area;

[0038] Inputting the mask image and the preset noise image into a first-stage diffusion model, and performing denoising processing on the preset noise image based on the mask image by the first-stage diffusion model to obtain a low-resolution synthetic image including a target object having a region shape consistent with the mask region;

[0039] The low-resolution synthetic image is input into a second-stage diffusion model cascaded with the first-stage diffusion model, and the low-resolution synthetic image is denoised by the second-stage diffusion model to obtain a high-resolution synthetic image corresponding to the low-resolution synthetic image; the high-resolution synthetic image is used for image segmentation model training; and the image segmentation model is used for image segmentation.

[0040] The above-mentioned image segmentation method, apparatus, computer device, computer-readable storage medium and computer program product based on data generation can obtain a mask image, input the mask image into the relay diffusion model, and guide the relay diffusion model to generate a high-resolution synthetic image through the mask image. Since the high-resolution synthetic image is generated by a machine learning model, it does not involve issues such as user privacy and ownership. Therefore, the high-resolution synthetic image is used as a compliant model training sample, and then a compliant synthetic data set for training the model can be obtained based on multiple high-resolution synthetic images. The image segmentation model is trained based on the compliant data set, and image segmentation is performed based on the trained image segmentation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0042] FIG1 is a diagram showing an application environment of an image segmentation method based on data generation in one embodiment;

[0043] FIG2 is a schematic diagram of a flow chart of an image segmentation method based on data generation in one embodiment;

[0044] FIG3 is a schematic diagram of generating a composite image in one embodiment;

[0045] FIG4 is a schematic diagram of an overall framework for training an image segmentation model in one embodiment;

[0046] FIG5 is a structural block diagram of an image segmentation device based on data generation in one embodiment;

[0047] FIG6 is a diagram showing the internal structure of a computer device in one embodiment;

[0048] FIG. 7 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] The image segmentation method based on data generation provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. Among them, the client 102 communicates with the server 104 through the network. The data storage system can store the data that the client 102 needs to process. The computer device 102 can be a server or a terminal, and the data storage system can be integrated on the server, or it can be placed on the cloud or other network servers. A relay diffusion model is deployed on the client, wherein the relay diffusion model includes a first-stage diffusion model and a second diffusion model connected to the first-stage diffusion model. The first-stage diffusion model is combined with an additional input channel, and the additional input channel is used to input a mask image. The relay diffusion model in the client 102 can generate a synthetic image based on the input mask image, and send the mask image and the synthetic image together to the server 104. When the server 104 receives the synthetic image and mask image returned by each client 102, a training sample set can be established based on the received synthetic image and mask image. The client 102 includes but is not limited to various desktop computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0051] In an exemplary embodiment, as shown in FIG2 , a method for image segmentation based on data generation is provided, which is described by taking the method applied to a client as an example, and includes the following steps 202 to 206. Among them:

[0052] Step 202, obtaining a mask image; the mask image includes a mask area.

[0053] Specifically, the client may obtain a mask image. The mask image includes a mask area and a non-mask area. For example, the mask image may include a mask area and a non-mask area. The mask area and the non-mask area may be distinguished by pixel values. For example, the pixel value of a pixel point in the mask area may be (255, 255, 255), and the pixel value of a pixel point in the non-mask area may be (0, 0, 0).

[0054] Step 204, input the mask image and the preset noise image into the first stage diffusion model, and perform denoising on the preset noise image based on the mask image through the first stage diffusion model to obtain a low-resolution composite image including the target object having a region shape consistent with the mask region.

[0055] Specifically, the client inputs the mask image and the noise image into the first-stage diffusion model. The first-stage diffusion model adds an additional mask image input channel on the basis of the traditional diffusion model, which is used to receive the mask image, so as to guide the first-stage diffusion model to output a low-resolution synthetic image through the mask image. Among them, the low-resolution synthetic image includes a low-resolution synthetic image of the target object that matches the regional shape of the mask area. For example, referring to Figure 3, Figure 3 shows a schematic diagram of generating a synthetic image in an embodiment. As shown in Figure 3, a 64*64 image can be obtained through the first-stage diffusion model.

[0056] In one embodiment, the noise image may be an image including Gaussian distribution noise. The image size of the noise image may be consistent with the image size of the mask image. The area in the noise image that is masked by the mask area is the target area. The diffusion model may perform multiple rounds of denoising on the noise image with the target area as a condition to obtain the target object in the low-resolution synthetic image.

[0057] The first-stage diffusion model can perform multiple rounds of denoising on the preset noise image based on the mask image to obtain a low-resolution synthetic image.

[0058] Step 206, input the low-resolution synthetic image to the second-stage diffusion model cascaded with the first-stage diffusion model, perform denoising on the low-resolution synthetic image through the second-stage diffusion model, and obtain a high-resolution synthetic image corresponding to the low-resolution synthetic image; the high-resolution synthetic image is used for image segmentation model training; the image segmentation model is used for image segmentation.

[0059] Specifically, when a low-resolution synthetic image is obtained, the low-resolution synthetic image can be input into a second-stage diffusion model cascaded with the first-stage diffusion model, and the low-resolution synthetic image is denoised by the second-stage diffusion model to obtain a high-resolution synthetic image. The resolution of the high-resolution synthetic image is higher than that of the low-resolution synthetic image.

[0060] In one embodiment, a low-resolution synthetic image is input into a second-stage diffusion model cascaded with a first-stage diffusion model, and the first low-resolution image is denoised by the second-stage diffusion model to obtain a high-resolution synthetic image corresponding to the low-resolution synthetic image, including: upsampling the low-resolution synthetic image to obtain an upsampled image; and performing multiple rounds of denoising on the upsampled image by the second-stage diffusion model to obtain a high-resolution synthetic image corresponding to the low-resolution synthetic image.

[0061] Specifically, the low-resolution synthetic image can be upsampled to obtain an upsampled image, and the upsampled image can be subjected to multiple rounds of denoising to obtain a high-resolution synthetic image corresponding to the low-resolution synthetic image. For example, referring to FIG3 , a low-resolution synthetic image of 64*64 size can be upsampled to obtain an upsampled image of 256*256 size, and then the upsampled image of 256*256 size can be subjected to multiple rounds of denoising to obtain a high-resolution synthetic image of 256*256 size. The high-resolution synthetic image can be used for image segmentation. For example, the high-resolution synthetic image can be input into an image segmentation model to obtain an image segmentation result, and then the model parameters of the image segmentation model can be adjusted based on the image segmentation result.

[0062] It is easy to understand that the first-stage diffusion model and the second-stage diffusion model constitute a relay diffusion model. The method proposed in this scheme adopts a two-stage cascade diffusion structure. By modifying the first stage of the relay diffusion model, a mask conditional channel is introduced to form a mask-guided diffusion model, which can guide the generation of low-resolution images based on mask images. When generating high-resolution images, the relay diffusion model uses a noisy low-resolution image as the starting point instead of pure Gaussian noise, eliminating the part of generating low-frequency information, greatly saving computational costs, and is simpler, without the need for low-resolution images as conditions and various conditional enhancement techniques, and does not require the redesign or adjustment of the noise schedule.

[0063] In one embodiment, before outputting a high-resolution synthetic image through the relay diffusion model, the relay diffusion model needs to be trained. The client can obtain a mask sample image and a training sample image, and input the mask sample image and the training sample image into the relay diffusion model. The training sample image is subjected to multiple rounds of noise processing through the relay diffusion model to obtain a noise sample image, and the mask sample image is used to guide the relay diffusion model to perform multiple rounds of denoising processing on the noise sample image. In each denoising process, the noise prediction network of the relay diffusion model can predict a noise, so as to adjust the model parameters of the relay diffusion model based on the difference between the predicted noise and the standard noise.

[0064] In one implementation, the original diffusion model can only be derived from the unconditional data distribution The goal of this embodiment is to conditionally generate an image with a c channel. , so that it conforms to a mask sample image , where C is the number of classes (including background), h is the height of the image, and w is the width of the image. That is, the class of this mask sample image may be 0, ..., C-1. In other words, the traditional diffusion model can only be derived from the unconditional data distribution. The relay diffusion model in this embodiment can be sampled from the conditional distribution Although the data likelihood is modified to the form of mask guidance It does not change the forward noise addition process. , but will modify the reverse process and noise prediction network Therefore, the loss function of the relay diffusion model is as follows:

[0065]

[0066] in, is the standard noise, is the noise predicted by the noise prediction network in the relay diffusion model. m is the mask sample image. t represents the tth time (that is, the tth round mentioned above). This is the output image of the relay diffusion model. , . is the weight term at time t.

[0067] It should be emphasized that under the framework of multi-center image segmentation, it is practical for the client to use mask images to train the relay diffusion model, and it has some unique advantages: (1) Fast, the model convergence speed is much higher than that of the ordinary diffusion model structure, and it can still work well on edge devices (i.e., clients) with limited computing resources. (2) Easy to train and stable, no complex parameter adjustment is required, and it can still produce synthetic images with stable quality under small samples. This greatly reduces the limitations of the client using the relay diffusion model. (3) Easy to expand, the relay diffusion model can be started from images of any resolution and easily expanded to higher resolutions.

[0068] In the above-mentioned image segmentation method based on data generation, by obtaining a mask image, the mask image can be input into the relay diffusion model, and the mask image is used to guide the relay diffusion model to generate a high-resolution synthetic image. Since the high-resolution synthetic image is generated by a machine learning model, it does not involve issues such as user privacy and ownership. Therefore, the high-resolution synthetic image is used as a compliant model training sample, and then a compliant synthetic data set for training the model can be obtained based on multiple high-resolution synthetic images, so that the high-resolution synthetic image can be input into the image segmentation model for image segmentation, and the model parameters of the image segmentation model can be adjusted based on the image segmentation results.

[0069] In one of the embodiments, a preset noise image is denoised based on a mask image through a first-stage diffusion model to obtain a low-resolution synthetic image including a target object having a regional shape that matches the mask area, including: determining a target area corresponding to the mask area in a preset noise image; and performing multiple rounds of denoising on the preset noise image based on the target area to obtain a low-resolution synthetic image including a target object having a regional shape that matches the mask area.

[0070] Specifically, the mask image can be used to guide the editing of the target area pointed to by the mask area in the cross-attention layer of the first-stage diffusion model, thereby obtaining a low-resolution synthetic image including the target object having a regional shape consistent with the mask area.

[0071] In one embodiment, a description text may be input, and the description text is used to describe the type of the target object. For example, the description text may be "generate cancer cells", so that the obtained low-resolution image may include cancer cells, and the cancer cells are in the target area indicated by the mask image.

[0072] In one embodiment, the original image can be obtained, and the original image can be subjected to multiple rounds of noise addition processing to obtain a noise image. During forward diffusion, the noise image after each step of noise addition is recorded. Forward noise addition and backward denoising are equal-length processes, that is, the number of noise addition rounds in the forward noise addition process is consistent with the number of denoising rounds in the backward denoising process. Therefore, each noise addition round can correspond to each denoising round one by one, for example, the first denoising round can correspond to the last noise addition round. The information of the original data is gradually lost during forward noise addition, while the information is gradually supplemented from pure noise during backward denoising. If the noise image of each step in the forward noise addition process is recorded and mixed with the noise image in the backward process, the generation result of the backward process can be affected. For example, in the backward denoising process, for the denoising process of the current round, the noise addition process corresponding to the denoising process of the current round is determined, and the noise image generated in the noise addition process corresponding to the denoising process of the current round is determined. In the backward denoising process, the unmasked area in the image output in the previous round is changed to the unmasked area in the noisy image generated in the corresponding denoising process, and the changed image is subjected to the denoising process in the current round. This process is repeated until the last round of denoising process, and a low-resolution synthetic image is obtained.

[0073] In one of the embodiments, the above method also includes: sending the high-resolution composite image and the mask image to the server, so that the server generates a composite data set based on the high-resolution composite image and the mask image sent by each client, training the image segmentation model to be trained based on the composite data set, and returning the trained image segmentation model to each client.

[0074] Specifically, the mask image and the high-resolution synthetic image generated based on the mask image can be used as an image pair. After the client generates multiple image pairs based on the relay diffusion model, the multiple image pairs can be sent to the server in batches. The server can receive the image pairs sent by each client and generate a synthetic data set based on the received image pairs. For example, for each client, the server can sample from the image pairs sent in batches by the client, and store the sampled image pairs in a set. In this way, when the image pairs sampled from the image pairs sent in batches by each client are stored in the same set, the set can be used as a synthetic data set.

[0075] Furthermore, the server may train the image segmentation model to be trained based on the synthetic data set. For example, the high-resolution synthetic image in the image pair may be used as a training sample, and the mask image in the image pair may be used as a label. The high-resolution synthetic image is segmented by the image segmentation model to be trained to obtain a predicted image segmentation result. Based on the difference between the predicted image segmentation result and the mask image, the model parameters of the image segmentation model are adjusted until the training stop condition is reached, thereby obtaining a trained image segmentation model.

[0076] Furthermore, after the server obtains the trained image segmentation model, the server can send the trained image segmentation model to each client, so that each client can use the image segmentation model. The above process can be called MCMS-Aug-Syn.

[0077] In this embodiment, during this process, the client only needs to upload the image pairs output by the cascade diffusion model trained based on the local data set. The server performs sampling, generates a synthetic data set and trains the image segmentation model without any additional communication or calculation. Compared with the traditional federated learning that requires multiple nodes to participate in multiple rounds of parameter or gradient iterations, this embodiment does not require multiple nodes to participate in multiple rounds of parameter or gradient iterations, thus saving communication resources between the client and the server. In addition, the participation of multiple nodes in traditional federated learning in multiple rounds of parameter or gradient iterations will also restrict the training speed of the model. In this embodiment, the image segmentation model can be trained directly in the server, thus improving the model training efficiency. In addition, since the segmentation model does not need to be trained on the client, the local computing cost is greatly reduced in this embodiment.

[0078] In one of the embodiments, an image segmentation model to be trained is trained based on a synthetic data set, including, for each mask image in the synthetic data set, adjusting the pixel values ​​of pixels on a boundary area in the mask image based on a preset smoothing parameter to obtain a mask image with adjusted pixel values; and using the mask image with adjusted pixel values ​​as a label to train the image segmentation model to be trained.

[0079] Specifically, when it is necessary to train an image segmentation model based on a synthetic dataset, the synthesis process may introduce bias due to non-uniform sampling of the data manifold when training the relay diffusion model. This leads to uncertainty in the labels of the boundaries of the segmented target objects in the generated high-resolution synthetic images, and some pixels may be mislabeled, which limits the performance of the image segmentation model trained on the synthetic dataset. To solve this problem, the mask image is processed in the synthetic dataset to form a soft label, which prevents the image segmentation model from being overconfident about the labels in the synthetic dataset based on the soft label.

[0080] More specifically, for each image pair in the synthetic data set, the server can obtain a preset smoothing parameter, and adjust the pixel values ​​on the boundary area of ​​the mask image in the image pair by the smoothing parameter to obtain the mask image after the pixel value is adjusted. For example, the smoothing coefficient can be 0.1, and for each pixel point in the mask image, the server can adjust the pixel value of the pixel point, and the adjustment range can be between +0.1 and -0.1. The boundary area refers to the connection between the mask area and the non-mask area.

[0081] In one embodiment, the connection between the masked area and the non-masked area in the mask image corresponds to the standard segmentation boundary of the image segmentation. Therefore, the pixel points at the connection between the masked area and the non-masked area need to be treated with emphasis. For example, the pixel value of each pixel point at the connection between the masked area and the non-masked area can be adjusted. For the non-connected area, the pixel points can be randomly selected and the pixel values ​​of the randomly selected pixel points can be adjusted. In this way, the amount of pixel adjustment can be reduced, saving computer resources and other resources consumed when adjusting the pixel points.

[0082] In one embodiment, during the training of the image segmentation model, the following loss function may be used to adjust the model parameters of the image segmentation model:

[0083]

[0084] Where x is the predicted image segmentation result output by the image segmentation model (that is, the predicted mask image), and y is the mask image after pixel value adjustment.

[0085] In the above embodiment, a mask image with adjusted pixel values ​​is generated for a high-resolution synthetic image, and a loss function is obtained. The performance of the synthetic data set in training can be enhanced based on the loss function, and the gap between the synthetic data set and the real data set can be further narrowed, thereby improving the accuracy of image segmentation model training based on the synthetic data set with a reduced gap. Regardless of the model structure used to train the image segmentation model in decentralized learning, this module can be easily integrated to enhance the training of synthetic data.

[0086] In one of the embodiments, the above method also includes: sending the high-resolution composite image and the mask map to the server, so that the server generates a composite data set based on the high-resolution composite image and the mask map sent by each client; receiving the composite data set returned by the server, and training the image segmentation model to be trained based on the preset local data set and the composite data set; performing image segmentation on the segmented image by using the trained image segmentation model to obtain a segmentation result, and returning the segmentation result to the server, so that the server performs integrated learning based on the segmentation results sent by each client to obtain an integrated result.

[0087] Specifically, after the client generates a high-resolution composite image, the mask image and the high-resolution composite image generated based on the mask image can be sent to the server as an image pair. The server can receive one or more image pairs sent by each client, sample the received image pairs, and store the sampled image pairs in a set to obtain a composite data set. Further, the server sends the composite data set to each client. For each client, the client can train the image segmentation model to be trained based on the preset local data set and the composite data set. In this way, each client can train an image segmentation model. Among them, the way the client trains the image segmentation model is consistent with the way the server trains the image segmentation model, except that the client can also be trained in combination with the local data set.

[0088] Furthermore, each client can perform image segmentation processing on the same image to be segmented based on the trained image segmentation model to obtain a segmentation result, and return the segmentation result to the server. In this way, the server can integrate the segmentation results returned by each image segmentation model to obtain an integrated result. For example, the segmentation result can be a predicted mask image output by the image segmentation model. The server can average the predicted mask images returned by each client to obtain an integrated result. The averaging process can be to superimpose the pixel values ​​of the pixels at the same position in each predicted mask image to obtain a superimposed pixel value, and divide the superimposed pixel value by the number of images of the predicted mask image to obtain the pixel mean at the position, and obtain the integrated result based on the pixel mean at each position. The above process can be called MCMS-Aug-Ens. Since there are differences between the local data sets of each client, there are differences between the image segmentation models trained in combination with the local data sets. The integrated result obtained in the integration is usually more accurate than the result output by any single locally trained image segmentation model.

[0089] In one embodiment, the integration result can be obtained by the following formula:

[0090]

[0091] Where T is the total number of predicted mask images; is the pixel value of the pixel at position x in the i-th predicted mask image.

[0092] In one embodiment, each client may also send the model parameters of the trained image segmentation model to the server, so that the server averages the model parameters to obtain the mean parameters, and then uses the mean parameters as the model parameters of the image segmentation model to be trained, thereby obtaining an aggregate model. Since there are differences between the local data sets of each client, the image segmentation models obtained by combining the local data sets for training are different, and the obtained aggregate model can usually achieve better performance than any single local trained image segmentation model.

[0093] In the above embodiment, two rounds of communication are required between the client and the server. It is worth noting that this embodiment supports and encourages the client to train segmentation models with different architectures to enhance the diversity and effectiveness of the integration. In addition, since private local data and shared synthetic data are used to train the image segmentation model, the performance of the trained image segmentation model can be very close to the performance of the image segmentation model obtained by centralized learning.

[0094] In one embodiment, after the server generates a synthetic data set, the synthetic data set may be redistributed to each client. The synthetic data set is used by the client to train the image segmentation model. After the training is completed, the client does not need to upload the trained image segmentation model, but uploads the image segmentation result of the trained image segmentation model on the synthetic data set, and sends the image segmentation result on the synthetic data set to the server. The server may integrate the received image segmentation results to obtain an integrated result, and use the integrated result as a teacher to train the student model through knowledge distillation. For example, the integrated result may be used as a label to train the student model. The above process may be referred to as MCMS-Aug-KD.

[0095] It is easy to understand that when the resources of edge devices are limited, the method of "obtaining the image segmentation model by training the image segmentation model through the server" can be preferred. For scenarios that require higher accuracy, the method of "training the image segmentation model through the client" can be adopted. When the client uses a unique model architecture or commercial model and does not want to share model information with other participants in multi-party learning, the method of "the server receives the results of each image segmentation, and obtains an integrated result based on the image segmentation results, uses the integrated result as a teacher, and trains the student model through knowledge distillation" can be adopted. Compared with federated learning, the proposed method has significant advantages in communication cost and can be integrated with various popular segmentation baselines. They can achieve excellent performance on different datasets without complex parameter adjustment. In addition, this method can not only generate a global image segmentation model, but also build a secure and shareable synthetic dataset, enabling clients to use synthetic data to enhance various downstream tasks.

[0096] In one embodiment, referring to FIG. 4, FIG. 4 shows a schematic diagram of the overall framework of training an image segmentation model in one embodiment. The training of the image segmentation model is divided into three stages, (1) an image pair generation module, referred to as a data generation stage; (2) a label optimization module, referred to as a segmentation training stage; and (3) an integration and distillation module, referred to as a model aggregation stage. In the image pair generation module, each client can first train the relay diffusion model to obtain a trained relay diffusion model. The mask image output by the mask generator and the preset noise image are input into the trained relay diffusion model, and a high-resolution synthetic image is output through the trained relay diffusion model. The high-resolution synthetic image and the mask image are used as an image pair, and then the server can obtain a synthetic data set based on the image pairs generated by each client. In the second stage, the mask image used for training the image segmentation model can be subjected to pixel value adjustment processing to obtain a mask image after pixel value adjustment, and then the image segmentation model is trained based on the mask image after pixel value adjustment. The training of the image segmentation model can be performed by the client, or it can be performed by the server. In the third stage, after obtaining the trained image segmentation model, the image segmentation models can be integrated to obtain a global image segmentation model, or the segmentation results output by each image segmentation model can be integrated to obtain an integrated result, and the image segmentation model to be trained can be trained based on the integrated result to obtain a global image segmentation model.

[0097] The image pair generation method based on the relay diffusion model proposed in this application can still learn the data distribution of a specific client and stably generate high-quality samples under limited data and computing resources. The generated annotated samples can form a large public synthetic dataset to facilitate downstream image segmentation tasks.

[0098] The mask image pixel value adjustment method proposed in this application further improves the performance of synthetic images in downstream image segmentation model training tasks. It has been demonstrated on five image segmentation datasets that when the image segmentation model is trained using only synthetic images and mask images, the performance far exceeds that of a single client dataset and existing federated learning methods, and is close to centralized learning of all clients. If the image segmentation model is trained on each client using synthetic images and mask image data with adjusted pixel values, and the image segmentation models trained on each client are aggregated using an integrated distillation method, the performance of the aggregated image segmentation model even exceeds the centralized learning of all clients.

[0099] The distributed image segmentation framework proposed in this application is highly practical and has no restrictions on the structure of the client model. It can be easily combined with the existing leading segmentation baseline model. In addition to generating the model, the client does not need to upload additional information or perform multiple rounds of communication and training. It can also ensure that the method is simple and effective enough to achieve superior performance on various data sets without any complex parameter adjustments, so as to promote its use among clients and promote knowledge sharing.

[0100] In an exemplary embodiment, as shown in FIG5 , a data generation-based image segmentation device 500 is provided, comprising: an image acquisition module 501, a low-resolution image generation module 502, and a high-resolution image generation module 503, wherein:

[0101] The image acquisition module 501 is used to acquire a mask image; the mask image includes a mask area;

[0102] A low-resolution image generation module 502 is used to input the mask image and the preset noise image into the first-stage diffusion model, and perform denoising on the preset noise image based on the mask image through the first-stage diffusion model to obtain a low-resolution synthetic image including a target object having a region shape consistent with the mask region;

[0103] The high-resolution image generation module 503 is used to input the low-resolution synthetic image into the second-stage diffusion model cascaded with the first-stage diffusion model, and perform denoising on the low-resolution synthetic image through the second-stage diffusion model to obtain a high-resolution synthetic image corresponding to the low-resolution synthetic image; the high-resolution synthetic image is used for image segmentation model training; and the image segmentation model is used for image segmentation.

[0104] In one of the embodiments, the low-resolution image generation module 502 is also used to determine the target area corresponding to the mask area in the preset noise map; based on the target area, the preset noise map is subjected to multiple rounds of denoising processing to obtain a low-resolution composite image including a target object having a regional shape consistent with the mask area.

[0105] In one embodiment, the high-resolution image generation module 503 is also used to upsample the low-resolution synthetic image to obtain an upsampled image; and perform multiple rounds of denoising on the upsampled image through the second-stage diffusion model to obtain a high-resolution synthetic image corresponding to the low-resolution synthetic image.

[0106] In one embodiment, the synthetic image generating device 500 also includes an image segmentation model training module, which is used to send the high-resolution synthetic image and the mask image to the server, so that the server generates a synthetic data set based on the high-resolution synthetic image and the mask image sent by each client, trains the image segmentation model to be trained based on the synthetic data set, and returns the trained image segmentation model to each client.

[0107] In one embodiment, the synthetic image generating device 500 also includes an image segmentation model training module, which is used to adjust the pixel values ​​of the pixel points on the boundary area of ​​the mask image based on a preset smoothing parameter for each mask image in the synthetic data set to obtain a mask image with adjusted pixel values; and use the mask image with adjusted pixel values ​​as a label to train the image segmentation model to be trained.

[0108] In one embodiment, the synthetic image generating device 500 also includes an integration module, which is used to send the high-resolution synthetic image and the mask map to the server, so that the server generates a synthetic data set based on the high-resolution synthetic image and the mask map sent by each client; receives the synthetic data set returned by the server, and trains the image segmentation model to be trained based on a preset local data set and the synthetic data set; performs image segmentation on the segmented image through the trained image segmentation model to obtain a segmentation result, and returns the segmentation result to the server, so that the server performs integrated learning based on the segmentation results sent by each of the clients to obtain an integrated result.

[0109] Each module in the above-mentioned image segmentation device based on data generation can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0110] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for image segmentation based on data generation is implemented.

[0111] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG7. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image segmentation method based on data generation is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0112] Those skilled in the art will understand that the structures shown in FIGS. 6 to 7 are merely block diagrams of partial structures related to the scheme of the present application, and do not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

[0113] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0114] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0115] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0116] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0117] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0118] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0119] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image segmentation method based on data generation, It is characterized in that Applied to a client, the method comprises: Acquire a mask image; the mask image includes a mask area; Inputting the mask image and the preset noise image into a first-stage diffusion model, and performing denoising processing on the preset noise image based on the mask image by the first-stage diffusion model to obtain a low-resolution synthetic image including a target object having a region shape consistent with the mask region; The low-resolution synthetic image is input into a second-stage diffusion model cascaded with the first-stage diffusion model, and the low-resolution synthetic image is denoised by the second-stage diffusion model to obtain a high-resolution synthetic image corresponding to the low-resolution synthetic image; the high-resolution synthetic image is used for image segmentation model training; and the image segmentation model is used for image segmentation.

2. The method according to claim 1, It is characterized in that The denoising process is performed on the preset noise image based on the mask image by using the first-stage diffusion model to obtain a low-resolution synthetic image including a target object having a region shape consistent with the mask region, including: Determine a target area corresponding to the mask area in the preset noise map; Taking the target area as a condition, the preset noise map is subjected to multiple rounds of denoising processing to obtain a low-resolution synthetic image including the target object having a regional shape consistent with the mask area.

3. The method according to claim 1, It is characterized in that The step of inputting the low-resolution synthetic image to a second-stage diffusion model cascaded with the first-stage diffusion model, and performing denoising on the low-resolution synthetic image by the second-stage diffusion model to obtain a high-resolution synthetic image corresponding to the low-resolution synthetic image comprises: Performing upsampling processing on the low-resolution synthetic image to obtain an upsampled image; The upsampled image is subjected to multiple rounds of denoising processing through the second-stage diffusion model to obtain a high-resolution synthetic image corresponding to the low-resolution synthetic image.

4. The method according to claim 1, It is characterized in that The method further comprises: The high-resolution composite image and the mask image are sent to a server, so that the server generates a composite data set based on the high-resolution composite image and the mask image sent by each client, trains the image segmentation model to be trained based on the composite data set, and returns the trained image segmentation model to each client.

5. The method according to claim 4, It is characterized in that The training of the image segmentation model to be trained based on the synthetic data set includes: For each mask image in the synthetic data set, adjusting the pixel values ​​of the pixel points on the boundary area of ​​the mask image based on a preset smoothing parameter to obtain a mask image with adjusted pixel values; The mask image with adjusted pixel values ​​is used as a label to train the image segmentation model to be trained.

6. The method according to claim 1, It is characterized in that The method further comprises: Sending the high-resolution synthetic image and the mask map to a server, so that the server generates a synthetic data set based on the high-resolution synthetic image and the mask map sent by each client; Receiving the synthetic data set returned by the server, and training the image segmentation model to be trained based on the preset local data set and the synthetic data set; The image with segmentation is segmented by using the trained image segmentation model to obtain a segmentation result, and the segmentation result is returned to the server, so that the server performs integrated learning based on the segmentation results sent by each of the clients to obtain an integrated result.

7. An image segmentation device based on data generation, It is characterized in that The device comprises: An image acquisition module, used to acquire a mask image; the mask image includes a mask area; a low-resolution image generation module, configured to input the mask image and the preset noise image into a first-stage diffusion model, and perform denoising processing on the preset noise image based on the mask image through the first-stage diffusion model to obtain a low-resolution synthetic image including a target object having a region shape consistent with the mask region; A high-resolution image generation module is used to input the low-resolution synthetic image into a second-stage diffusion model cascaded with the first-stage diffusion model, and perform denoising on the low-resolution synthetic image through the second-stage diffusion model to obtain a high-resolution synthetic image corresponding to the low-resolution synthetic image; the high-resolution synthetic image is used to train an image segmentation model; and the image segmentation model is used to perform image segmentation.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images