Method and apparatus for high resolution data synthesis for bipartite image segmentation

Abstract

The application discloses a high-resolution data synthesis method for binary image segmentation, compared with the prior art, the method generates high-quality and high-resolution synthetic data based on a two-stage method of mask editing and image generation, significantly improving the authenticity and detail fidelity of the synthetic data; wherein the mask editing uses rigid and non-rigid editing to generate diversified synthetic masks, effectively expanding the scale of the training data, the topological preservation adversarial training in the non-rigid editing ensures that the synthetic masks remain consistent in overall structure while changing locally, improving the quality of the synthetic data; the image generation adopts a multi-condition control generation method to ensure that the generated high-resolution images accurately correspond to the synthetic masks, meet the pixel-level accuracy requirements of the binary image segmentation task, and improve the generalization ability of the model. The application also provides a high-resolution data synthesis device for binary image segmentation and high-resolution data synthesis for binary image segmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and particularly relates to a high-resolution data synthesis method and device for binary image segmentation. BACKGROUND

[0002] Binary image segmentation is a key task in the field of artificial intelligence, which aims to accurately separate target objects from the background in images. Binary image segmentation has wide applications in medical image analysis, autonomous driving, environmental monitoring, and many other fields.

[0003] However, the binary image segmentation task requires high accuracy of labeling, and traditional dataset creation methods face these challenges, including: 1) high cost: requires a large amount of manpower and time investment, especially for large-scale, high-resolution datasets; 2) high professional requirements: requires relevant professional knowledge to accurately label; 3) insufficient data diversity: manually labeled datasets are often difficult to cover all possible scenarios and changes. For example, reference 1 (Xuebin Qin, Hang Dai, et al. Highly Accurate Dichotomous Image Segmentation, arXiv-CS-Computer Vision and Pattern Recognition, Doi: arXiv: 2203.03041) collected more than 12,000 images manually, then selected 22 groups of 5,470 images covering 225 categories based on the structural complexity of objects using convolutional neural networks as the DIS dataset called DIS5K, covering various backgrounds of camouflage, prominent or detailed objects, and using feature-level and mask-level guidance for binary image segmentation. By manually collecting images to construct a binary image segmentation dataset, a large amount of cost investment is required, and it is difficult to cover all possible scenarios and changes.

[0004] In recent years, using synthetic data to augment training sets has become a potential solution. For example, reference 2 (Quang Nguyen, Truong Vu, et al. Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentation, arXiv-CS-Computer Vision and Pattern Recognition, Doi: arxiv-2309.14303) proposes a method for generating synthetic datasets for pixel-level semantic segmentation based on diffusion. By using text-to-image generation model stable diffusion (SD) text prompts, cross-attention, and self-attention, three new technologies are introduced: class prompt attachment, class prompt cross-attention, and self-attention index. These technologies can generate segmentation maps corresponding to synthetic images, which can be used as pseudo labels to train semantic segmenters, eliminating the need for labor-intensive pixel-level annotations, and can merge uncertain areas into segmentation maps, thereby ignoring the loss of uncertain areas. Reference 3 (Weijia Wu, Yuzhong Zhao, et al. DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models, arXiv-CS-Computer Vision and Pattern Recognition, Doi: arxiv-2308.06160) proposes a general dataset generation model DatasetDM, which extends text-guided image synthesis to the generation of perception data by building on pre-trained diffusion models. Different synthetic images and corresponding high-quality perception annotations (e.g., segmentation masks and depth) are obtained to create an infinite annotated dataset for training various perception models for downstream tasks.

[0005] Although these works can use synthetic data to augment training sets to some extent, there are still problems, including: 1) lack of controllability: text prompt-based generation methods may produce results that deviate from real scenes, especially when dealing with fine-grained labels; 2) poor accuracy: using intermediate representations such as attention maps may introduce noise, reducing the fidelity of generated masks; 3) limited diversity: samples generated by existing methods often lack sufficient variation, making it difficult to fully enhance the generalization ability of models; 4) difficulty in generating high-resolution: ensuring accurate correspondence between image-mask pairs while maintaining high resolution remains a challenge. SUMMARY

[0006] The application aims to provide a high-resolution data synthesis method and device for binary image segmentation, which generates a high-quality and high-resolution synthetic data set based on a two-stage method of mask editing and image generation, to meet the requirement of high annotation accuracy for binary image segmentation tasks while expanding the distribution range of training data.

[0007] To achieve the above-mentioned application purposes, the embodiment provides a high-resolution data synthesis method and device for binary image segmentation, which includes a high-resolution data synthesis method for binary image segmentation and a device for implementing the high-resolution data synthesis method.

[0008] In one embodiment, the high-resolution data synthesis method for binary image segmentation includes the following steps:

[0009] Obtaining an original data set, wherein the original data set includes original images and segmentation masks corresponding to the original images;

[0010] Performing rigid editing and non-rigid editing on the segmentation masks corresponding to the original images in the original data set to obtain an edited synthetic mask set, wherein the edited synthetic mask set includes a rigidly edited synthetic mask set and a non-rigidly edited synthetic mask set;

[0011] Based on the edited synthetic mask set, using a multi-condition control generation method to generate corresponding high-resolution images;

[0012] Combining the edited synthetic mask set and the corresponding high-resolution images to obtain a high-resolution synthetic data set for binary image segmentation.

[0013] In one embodiment, the rigid editing uses one or more of rotation, translation, and scaling editing operations.

[0014] In one embodiment, the rigid editing includes the following steps:

[0015] (1) For each segmentation mask I, randomly generate a relative camera rotation and translation T of a target view angle i , wherein the number of segmentation masks is i~m;

[0016] (2) Invert the segmentation mask to obtain an inverted image

[0017] (3) Use a pre-trained view angle condition diffusion model ψ θ to generate a rigidly edited synthetic mask, taking the inverted image and the translation T i as inputs.

[0018] (4) Repeat steps (1)-(3) to obtain multiple rigid editing composite masks, and construct a rigid editing composite mask set.

[0019] In one embodiment, the non-rigid editing described employs one or more of local deformation, topology, scaling editing operations.

[0020] In one embodiment, the non-rigid editing described, the steps include:

[0021] (1) Construct a generator network G with parameters θ θ and a discriminator network D with parameters φ φ ;

[0022] (2) For the segmentation mask generate random Gaussian noise z and the corresponding text prompt p;

[0023] (3) Use the generator network G θ to generate a non-rigid editing composite mask based on the random Gaussian noise z, the text prompt p, and the segmentation mask ;

[0024] (4) Use a cross-attention mechanism to align the query feature Q of the non-rigid editing composite mask with the key-value feature (K, V) of the segmentation mask , where K represents the key feature and V represents the value feature, extract the composite mask M from the cross-attention graph to guide the focus of the model;

[0025] (5) Optimize the generator network G θ and the discriminator network D φ through topology-preserving adversarial training to construct a non-rigid editing composite mask set.

[0026] In one embodiment, the topology-preserving adversarial training includes:

[0027] (1) Apply an edge detection operator E to the segmentation mask and the non-rigid editing composite mask, respectively, to extract a key point set V , where v j represents each key point, and the number of key points is j ~ N v ;

[0028] (2) Based on the key point set V and the edge detection operator E, construct a topology structure graph T = (V, E);

[0029] (3) Use the discriminator network D φ to perform adversarial training on the segmentation mask topology structure graph T s and the composite mask topology structure graph T g .

[0030] In one embodiment, the optimization generator network G θ and the discriminator network D φ comprise:

[0031] The optimization generator network G θ and the discriminator network D φ comprise:

[0032] The discriminator network D φ is trained to maximize:

[0033]

[0034] where T s represents a segmentation mask topology graph, p represents a text prompt, T s ~p data (T s ) represents a segmentation mask topology graph based on the text prompt p, T g represents a non-rigid edit composite mask topology graph, T g ~p gen (T g ) represents a non-rigid edit composite mask topology graph based on the text prompt p.

[0035] The generator network G θ is trained to minimize:

[0036]

[0037] where T g represents a non-rigid edit composite mask topology graph, T g ~p gen (T g ) represents a non-rigid edit composite mask topology graph based on the text prompt p.

[0038] In one embodiment, the multi-condition control generation method uses edited composite masks as segmentation conditions, edge information as structure conditions, and text descriptions as semantic conditions.

[0039] In one embodiment, the multi-condition control generation method comprises the steps of:

[0040] (1) using edited composite masks as segmentation conditions where the number of edited composite masks is i~m, and the number of segmentation conditions is i~s;

[0041] (2) applying an edge detection operator to the edited composite masks to obtain edge conditions Wherein the number of edge conditions is i ~ y;

[0042] (3) The segmentation condition and the edge condition are input to the control block B0;

[0043] (4) The control block B θ outputs are injected into the pre-trained diffusion model M θ with parameters θ, control the denoising process of the random Gaussian noise z, and generate the corresponding high-resolution image

[0044]

[0045] In order to clearly show the high-resolution data synthesis method for binary image segmentation described above, the application also provides a high-resolution data synthesis device for binary image segmentation, comprising a memory for storing a computer program and a processor for implementing the high-resolution data synthesis method for binary image segmentation when executing the computer program.

[0046] Compared with the prior art, the application has at least the following beneficial effects:

[0047] The high-resolution data synthesis method for binary image segmentation provided by the application generates high-quality and high-resolution synthetic data based on a two-stage method of mask editing and image generation, significantly improving the authenticity and detail fidelity of the synthetic data. Among them, the mask editing uses rigid and non-rigid editing to generate diversified synthetic masks, effectively expanding the scale of the training data. The topological preservation adversarial training in non-rigid editing ensures that the synthetic mask remains consistent in overall structure while changing locally, improving the quality of the synthetic data. The multi-condition control generation method used in image generation ensures the accurate correspondence between the generated high-resolution image and the synthetic mask, meets the pixel-level accuracy requirements of the binary image segmentation task, and improves the generalization ability of the model.

[0048] The application also provides a high-resolution data synthesis device for binary image segmentation for binary image segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0050] Figure 1 A flowchart of a high-resolution data synthesis method for binary image segmentation.

[0051] Figure 2 A schematic diagram of effect display of original data, synthesized data after rigid editing and synthesized data after non-rigid editing.

[0052] Figure 3 A schematic diagram of effect display of original data and effect after processing by the high-resolution data synthesis method provided by the application. DETAILED DESCRIPTION

[0053] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be given to the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the protection scope of the present application.

[0054] In order to synthesize high-resolution image data for binary image segmentation tasks, the embodiment provides a high-resolution data synthesis method for binary image segmentation, which generates high-quality and high-resolution synthesized data based on a two-stage method of mask editing and image generation, significantly improves the authenticity and detail fidelity of the synthesized data and maintains the geometric alignment and multi-view consistency of the synthesized data.

[0055] The following will be described in combination with Figure 1 Specifically, the present application provides a high-resolution data synthesis method for binary image segmentation, which comprises the following steps:

[0056] S1, obtaining an original data set, wherein the original data set comprises original images and segmentation masks corresponding to the original images.

[0057] In the embodiment, the original data set is obtained wherein, the original images are the segmentation masks corresponding to the original images, wherein the number of original data is i~N, and the N original data constitutes an original data set, and the original data set can be any public or private binary image segmentation data set.

[0058] S2, performing rigid editing and non-rigid editing on the segmentation masks corresponding to the original images in the original data set to obtain an edited synthesized mask set, wherein the edited synthesized mask set comprises a rigid editing synthesized mask set and a non-rigid editing synthesized mask set.

[0059] In the embodiment, the segmentation masks in the original data set D are edited based on to obtain a rigid editing synthesized mask set wherein, denote each rigid editing composite mask, the number of rigid editing composite masks is i~M, and M rigid editing composite masks constitute a rigid editing composite mask set G m (rigid).

[0060] The rigid editing method includes but is not limited to rotation, translation, scaling and other editing operations that keep the overall structure of the mask unchanged, and the specific steps include:

[0061] (1) For each segmentation mask I, randomly generate a relative camera rotation and translation T of a target view i , where the number of segmentation masks is i~m. Here, the random generation can use uniform distribution or Gaussian distribution to determine the rotation angle and translation distance;

[0062] (2) Invert the segmentation mask to obtain the corresponding inverted image The inversion process can simply change the pixel value from 0 to 1 and from 1 to 0;

[0063] (3) Use the pre-trained view-conditioned diffusion model ψ θ , with the inverted image and the translation T i as input, to generate a rigid editing composite mask The diffusion model here can use an architecture similar to Stable Diffusion, but needs to additionally introduce a view condition to build a pre-trained view-conditioned diffusion model ψ θ ;

[0064] (4) Repeat the above steps until a sufficient number (M) of rigid editing composite masks are obtained to constitute a rigid editing composite mask set G m (rigid). Typically, multiple rigid editing composite masks of different views will be generated for each segmentation mask I to construct a rigid editing composite mask set, and the rigid editing composite data is as shown in Figure 2 .

[0065] In the embodiment, based on the segmentation mask I in the original data set D non-rigid editing is performed, and a non-rigid editing composite mask set G m (non- , where denote each non-rigid editing composite mask, the number of non-rigid editing composite masks is i~M, and M non-rigid editing composite masks constitute a non-rigid editing composite mask set G m (non-rigid).

[0066] The non-rigid editing method includes but is not limited to local deformation, topological change and other editing operations that can change the details of the mask, and the specific steps include:

[0067] (1) Construct a generator network G with a parameter θ θ and a discriminator network D with a parameter φ φ , wherein the generator network G θ adopts a U-Net structure, and the discriminator network D φ adopts a PatchGAN structure;

[0068] (2) For the segmentation mask , generate random noise z and the corresponding text prompt p, wherein the random noise z is usually sampled from a Gaussian distribution, and the text prompt p can be a natural language sentence describing the expected change, for example Figure 1 the modified shape prompt in the table: a round table;

[0069] (3) Use the generator network G θ to generate a non-rigid editing synthesized mask based on the random Gaussian noise z, the text prompt p and the segmentation mask ;

[0070] (4) Use the cross-attention mechanism to align the query feature Q of the non-rigid editing synthesized mask with the key-value feature (K, V) of the segmentation mask , so as to ensure that the non-rigid editing synthesized mask is consistent with the segmentation mask in the key area, wherein K represents the key feature and V represents the value feature, and then extract the synthesized mask M from the cross-attention diagram for subsequent loss calculation to guide the focus of the model;

[0071] (5) Optimize the generator network Gθ and the discriminator network D φ through topological preservation adversarial training, wherein the specific steps of the topological preservation adversarial training include:

[0072] (1) Apply an edge detection operator E to the segmentation mask and the non-rigid editing synthesized mask respectively to extract a key point set V of the topological structure, wherein v j represents each key point, the number of key points is j~N v , and N v key points constitute the key point set V; here the edge detection operator E can use a Canny edge detector or a Sobel operator;

[0073] (2) Based on the key point set V and the edge detection operator E, construct the topological structure graph T = (V, E) and use the k-nearest neighbor algorithm to determine the connection relationship of the edges;

[0074] (3) Using the discriminator network D φ For the segmentation mask topology diagram T s Non-rigid editing synthesis mask topology diagram T g Adversarial training is conducted to maintain topology consistency, whereby the discriminator network D... φ Parameterization is performed to maximize the training objective:

[0075]

[0076] Where T s This represents the topology diagram of the segmentation mask, where p represents the text prompt, and T represents the topology diagram of the segmentation mask. s ~p data (T s T represents the segmentation mask topology diagram based on text prompt p. g T represents the topology diagram of a non-rigid editable composite mask. g ~p gen (T g This represents a non-rigid edit composition mask topology diagram based on text prompts p;

[0077] For generator network G θ Parameterization to minimize the training objective:

[0078]

[0079] Where T g T represents the topology diagram of a non-rigid editable composite mask. g ~p gen (T g This represents the topology diagram of a non-rigid edit-synthesis mask based on text hints p. The non-rigid edit-synthesis mask set is constructed based on optimization, and the non-rigid edit-synthesis data is as follows: Figure 2 As shown.

[0080] S3. Based on the edited synthetic mask set, a multi-condition control generation method is used to generate the corresponding high-resolution image.

[0081] In the embodiment, based on the edited synthetic mask set G m A multi-condition control generation method is used to generate the corresponding high-resolution image set. in This represents the high-resolution image corresponding to the edited composite mask. The number of high-resolution images is i to M, and the M high-resolution images constitute a high-resolution image set G. r .

[0082] The multi-condition control generation method uses an edited synthetic mask set as a segmentation condition, edge information as a structural condition, and text description as a semantic condition. The specific steps include:

[0083] (1) Edit the composite mask As a dividing condition in Including rigid edit synthesis mask Non-rigid editing composite mask Segmentation conditions The number of elements is i to s;

[0084] (2) Edited composite mask Applying the Canny operator, we obtain the boundary conditions. Among the edge conditions The number of edges is i to y. The parameters of the Canny operator need to be adjusted according to the specific task to obtain the best edge information.

[0085] (3) The splitting conditions and edge conditions Input to control block B θ Control block B θ It could be a lightweight convolutional neural network used to process and fuse different conditional information;

[0086] (4) Control block B θ The output is injected into a pre-trained diffusion model M with parameters θ. θ In the process of controlling the denoising of random Gaussian noise z, a corresponding high-resolution image is generated.

[0087]

[0088] Among them, diffusion model M θ An architecture similar to Stable Diffusion can be used, but it needs to be adapted to inputs with multiple conditions controlled; repeat the above steps until all edited composite masks are obtained to generate the corresponding high-resolution images, and construct a high-resolution image set G. r .

[0089] S4. Combine the edited synthetic mask set with the corresponding high-resolution image to obtain a high-resolution synthetic dataset, which is used for binary image segmentation.

[0090] In the embodiment, the edited synthetic mask set G is... m and the corresponding high-resolution image set G r Combining them yields the final high-resolution synthetic dataset. in, represents final high-resolution synthetic data, the number of high-resolution synthetic data is i~M, and M high-resolution synthetic data constitutes a high-resolution synthetic data set G.

[0091] Figure 3 Some effects of the high-resolution data synthesis method for binary image segmentation are shown.

[0092] In the embodiment, a high-resolution data synthesis device for binary image segmentation is also provided, which comprises a memory and a processor, the memory is used to store a computer program, and the processor is used to realize the high-resolution data synthesis method for binary image segmentation when the computer program is executed.

[0093] The specific embodiments described above have explained the technical solutions and beneficial effects of the present application. It should be understood that the above description is only the most preferred embodiment of the present application and is not used to limit the present application. Any modification, supplement, and equivalent replacement within the principle range of the present application should be included in the protection range of the present application.

Claims

1. A high resolution data synthesis method for bipartite image segmentation, characterized in that, The method comprises the following steps: obtaining an original data set, wherein the original data set comprises original images and segmentation masks corresponding to the original images; performing rigid editing and non-rigid editing on the segmentation masks corresponding to the original images in the original data set to obtain an edited synthetic mask set, wherein the edited synthetic mask set comprises a rigid editing synthetic mask set and a non-rigid editing synthetic mask set; Based on the edited synthesis mask set, a multi-condition control generation method is used to generate the corresponding high-resolution image, wherein the multi-condition control generation method uses the edited synthesis mask set as the segmentation condition and the edge information as the structure condition, and specifically includes: (1) the edited synthesis mask is taken as the segmentation condition , wherein the number of the edited synthesis mask is i~m, and the number of the segmentation condition is i~s; (2) the edited synthesis mask is applied to the edge detection operator to obtain the edge condition , wherein the number of the edge condition is i~y; (3) the segmentation condition and the edge condition are input into a control block B θ ; (4) the output of the control block B θ is injected into a diffusion model M θ with pre-trained parameters θ, and the de-noising process of the random Gaussian noise z is controlled to generate the corresponding high-resolution image combining the edited synthetic mask set and the corresponding high-resolution images to obtain a high-resolution synthetic data set for binary image segmentation.

2. The method for high resolution data synthesis for bipartite image segmentation of claim 1, wherein, The rigid editing adopts one or more of rotation, translation, and scaling editing operations.

3. The method for high resolution data synthesis for bipartite image segmentation of claim 2, wherein, The rigid editing comprises the following steps: (1) For each segmentation mask I, randomly generate a target view's relative camera rotation and translation T i where the number of segmentation masks is i~m; (2) the segmentation mask is reversed to obtain a reversed image ​ (3) using a pre-trained view-conditioned diffusion model ψ θ to invert the image and translation T i as input, generating a rigid edit synthesis mask; (4) repeating steps (1) to (3) to obtain a plurality of rigid editing synthetic masks and construct a rigid editing synthetic mask set.

4. The method for high resolution data synthesis for bipartite image segmentation of claim 1, wherein, The non-rigid editing adopts one or more of local deformation and topological operations.

5. The method for high resolution data synthesis for bipartite image segmentation of claim 4, wherein, The non-rigid editing comprises the following steps: (1) construct a generator network G with parameters θ θ and a discriminator network D with parameters φ φ ; (2) for segmentation masks Generate random Gaussian noise z and corresponding text prompt p; (3) using a generator network G θ , based on random Gaussian noise z, a text prompt p, and a segmentation mask generates a non-rigid edit synthesis mask; (4) Adopting cross attention mechanism, aligning the query feature Q of the non-rigid editing synthesis mask with the key-value feature (K, V) pair of the segmentation mask, wherein K represents the key feature and V represents the value feature, extracting the synthesis mask M from the cross attention map to guide the focus of the model; ​ (5) Optimize the generator network G by topological preservation adversarial training θ and the discriminator network D φ , construct a non-rigid editing composite mask set.

6. The method for high resolution data synthesis for bipartite image segmentation of claim 5, wherein, The topological preservation adversarial training comprises: (1) to the segmentation mask and non-rigid editing synthesis mask respectively apply edge detection operator E, extract the key point set of topological structure where v j represent each key point, the number of key points is j~N v ; (2) constructing a topological structure graph T=(V,E) based on the key point set V and the edge detection operator E; (3) using a discriminator network D φ on the segmentation mask topology graph T s and the synthetic mask topology graph T g is adversarially trained.

7. The method for high resolution data synthesis for bipartite image segmentation of claim 5, wherein, The described optimisation generator network G θ and discriminator network D φ comprises: For the discriminator network D φ Training reaches a maximum: where T s represents a segmentation mask topology graph, p represents a text prompt, T s ~p data (T s ) represents a segmentation mask topology graph based on the text prompt p, T g represents a non-rigid edit composite mask topology graph, T g ~p gen (T g ) represents a non-rigid edit composite mask topology graph based on the text prompt p; For the generator network G θ Training reaches a minimum: where T g represents a non-rigid editing synthesis mask topology graph, T g ~p gen (T g ) represents a non-rigid editing synthesis mask topology graph based on text cue p.

8. A high resolution data synthesis device for bipartite image segmentation, comprising a memory and a processor, the memory being for storing a computer program, characterized in that, The processor is configured to implement the high-resolution data synthesis method for binary image segmentation according to any one of claims 1 to 7 when executing the computer program.

Images