High resolution data synthesis method and apparatus for bipartite image segmentation

Abstract

The invention discloses a high-resolution data synthesis method for bipartite image segmentation, and compared with an existing method, the method generates high-quality and high-resolution synthetic data based on a mask editing and image generation two-stage method, and significantly improves the authenticity and detail fidelity of the synthetic data. According to mask editing, rigid and non-rigid editing is used for generating diversified composite masks, the scale of training data is effectively expanded, topology in non-rigid editing keeps confrontation training, it is ensured that the composite masks locally change, meanwhile, the consistency of the overall structure is kept, and the quality of the composite data is improved; in the image generation process, a multi-condition control generation method is adopted, so that the accurate correspondence between the generated high-resolution image and the composite mask is ensured, the requirement of a bipartite image segmentation task on the pixel-level accuracy is met, and the generalization ability of the model is improved. The invention also provides a high-resolution data synthesis device for bipartite image segmentation, which is used for high-resolution data synthesis for bipartite image segmentation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a high-resolution data synthesis method and device for binary image segmentation. Background Art

[0002] Bisection image segmentation is a key task in the field of artificial intelligence, and its goal is to accurately separate the target object from the background in the image. Bisection image segmentation is widely used in many fields such as medical image analysis, autonomous driving, and environmental monitoring.

[0003] However, the task of binary image segmentation requires extremely high accuracy of annotation. Traditional dataset creation methods face these challenges, including: 1) high cost: a lot of manpower and time investment is required, especially for large-scale, high-resolution datasets; 2) high professional requirements: professional knowledge in related fields is required for accurate annotation; 3) insufficient data diversity: manually annotated datasets often find it difficult to cover all possible scenarios and changes. For example, Reference 1 (Xuebin Qin, Hang Dai, et al. Highly Accurate Dichotomous Image Segmentation, arXiv-CS-Computer Vision and Pattem Recognition, Doi: arXiv: 2203.03041) manually collected more than 12,000 images, and then selected them based on the structural complexity of the objects based on convolutional neural networks to obtain 5470 images covering 225 categories in 22 groups as the DIS dataset called DIS5K, covering camouflaged, prominent or detailed objects in various backgrounds, and using both feature-level and mask-level guidance for binary image segmentation. Constructing a binary image segmentation dataset by manually collecting images requires a lot of cost investment and it is difficult to cover all possible scenarios and changes.

[0004] In recent years, using synthetic data to expand the training set has become a potential solution. For example, reference 2 (Quang Nguyen, Truong Vu, et a1. Dataset Diffusion: Diffusion-based SyntheticDataset Generation for Pixel-Level Semantic Segmentation, arXiv-CS-ComputerVision and Pattem Recognition, Doi: arxiv-2309.14303) proposed a method for generating synthetic datasets for pixel-level semantic segmentation based on diffusion, using text hints, cross attention and self-attention of text hints of stable diffusion (SD) of text-to-image generation model, and introduced three new technologies: class hint addition, class hint cross attention and self-attention index to generate pixel-level semantic segmentation labels. These technologies can generate segmentation maps corresponding to synthetic images, and the mapping can be used as pseudo-labels for training semantic segmenters, eliminating the need for labor-intensive pixel-level annotations, and can merge uncertainty areas into segmentation maps, thereby ignoring the loss of uncertain areas. Reference 3 (Weijia Wu, Yuzhong Zhao, et al. DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models, arXiv-CS-Computer Vision and Pattern Recognition, Doi: arxiv-2308.06160) proposes a general dataset generation model DatasetDM, which extends text-guided image synthesis to the generation of perceptual data on top of a pre-trained diffusion model, with different synthetic images and corresponding high-quality perceptual annotations (e.g., segmentation masks and depth), to obtain an infinitely large annotated dataset for training various perceptual models for downstream tasks.

[0005] Although these works can use synthetic data to expand the training set to a certain extent, these problems still exist, including: 1) Insufficient controllability: Generation methods based on text prompts may produce results that deviate from the real scene, especially when dealing with fine-grained labels; 2) Poor accuracy: The use of intermediate representations such as attention maps may introduce noise and reduce the fidelity of the generated masks; 3) Diversity limitations: Samples generated by existing methods often lack sufficient variation, making it difficult to comprehensively improve the generalization ability of the model; 4) Difficulty in high-resolution generation: Ensuring the precise correspondence of image-mask pairs while maintaining high resolution remains a challenge. Summary of the invention

[0006] The purpose of the present invention is to provide a high-resolution data synthesis method and device for binary image segmentation, which generates a high-quality and high-resolution synthetic data set based on a two-stage method of mask editing and image generation, so as to meet the requirements of binary image segmentation tasks for high annotation accuracy while expanding the distribution range of training data.

[0007] To achieve the above-mentioned purpose of the invention, an embodiment provides a high-resolution data synthesis method and device for binary image segmentation, including a high-resolution data synthesis method for binary image segmentation and a device for implementing the high-resolution data synthesis method.

[0008] In one embodiment, the high-resolution data synthesis method for binary image segmentation comprises the following steps:

[0009] Obtain an original data set, where the original data set includes an original image and a segmentation mask corresponding to the original image;

[0010] Performing rigid editing and non-rigid editing on the segmentation mask corresponding to the original image in the original data set to obtain an edited synthetic mask set, wherein the edited synthetic mask set includes a rigidly edited synthetic mask set and a non-rigidly edited synthetic mask set;

[0011] Based on the edited synthetic mask set, a multi-conditional control generation method is used to generate the corresponding high-resolution image;

[0012] The edited synthetic mask set and the corresponding high-resolution image are combined to obtain a high-resolution synthetic dataset for bisection image segmentation.

[0013] In one embodiment, the rigid editing adopts one or more of rotation, translation, and scaling editing operations.

[0014] In one embodiment, the rigid editing step includes:

[0015] (1) For each segmentation mask I, randomly generate a relative camera rotation and translation T of the target view i , where the number of segmentation masks is i~m;

[0016] (2) The segmentation mask Perform inversion processing to obtain an inverted image

[0017] (3) Use the pre-trained view-conditional diffusion model ψ θ , to invert the image and pan T i As input, a rigid edit synthesis mask is generated;

[0018] (4) Repeat steps (1) to (3) to obtain multiple rigid edit synthesis masks and construct a rigid edit synthesis mask set.

[0019] In one embodiment, the non-rigid editing adopts one or more of local deformation, topology, and scaling editing operations.

[0020] In one embodiment, the non-rigid editing step includes:

[0021] (1) Construct a generator network G with parameter θ θ and a discriminator network D with parameter φ φ ;

[0022] (2) For segmentation mask Generate random Gaussian noise z and corresponding text prompt p;

[0023] (3) Using the generator network G θ , based on random Gaussian noise z, text hint p and segmentation mask Generate non-rigid edit synthesis mask;

[0024] (4) Using the cross-attention mechanism, the query feature Q of the non-rigid edit synthesis mask is combined with the segmentation mask The key-value features (K, V) of the proposed algorithm are aligned, where K represents the key feature and V represents the value feature. A synthetic mask M is extracted from the cross-attention map to guide the focus of the model.

[0025] (5) Optimize the generator network G through topology-preserving adversarial training θ and the discriminator network D φ ,Construct a set of non-rigid edit synthesis masks.

[0026] In one embodiment, the topology preserving adversarial training includes:

[0027] (1) Segmentation mask The edge detection operator E is applied to the non-rigid edit synthesis mask to extract the key point set of the topological structure. where v j Represents each key point, the number of key points ranges from j to N v ;

[0028] (2) Based on the key point set V and the edge detection operator E, a topological structure graph T = (V, E) is constructed;

[0029] (3) Using the discriminator network D φ For the segmentation mask topology graph T s And the composite mask topology structure diagram T g Conduct adversarial training.

[0030] In one embodiment, the optimization generator network G θ and the discriminator network D φ include:

[0031] The optimized generator network G θ and the discriminator network D φ include:

[0032] For the discriminator network D φ Training to the maximum:

[0033]

[0034] Where T s represents the segmentation mask topology, p represents the text prompt, T s ~p data (T s ) represents the segmentation mask topology graph based on the text hint p, T g represents the non-rigid edit synthesis mask topology graph, T g ~p gen (T g ) represents the non-rigid edit synthesis mask topology graph based on the text hint p;

[0035] For the generator network G θ Training is minimized:

[0036]

[0037] Where T g represents the non-rigid edit synthesis mask topology graph, T g ~p gen (T g ) represents the non-rigid edit synthesis mask topology graph based on the text hint p.

[0038] In one embodiment, the multi-conditional control generation method uses the edited synthetic mask set as the segmentation condition, the edge information as the structural condition, and the text description as the semantic condition.

[0039] In one embodiment, the multi-condition control generation method comprises the following steps:

[0040] (1) Edit the composite mask As a split condition The number of edited composite masks is i~m, and the number of segmentation conditions is i~s;

[0041] (2) Edited composite mask Apply edge detection operator to get edge condition The number of edge conditions is i~y;

[0042] (3) The split condition and edge conditions Input to control block Bθ;

[0043] (4) Control block B θ The output is injected into the pre-trained diffusion model M with parameters θ θ In the process of controlling the denoising of random Gaussian noise z, a corresponding high-resolution image is generated.

[0044]

[0045] In order to clearly demonstrate the high-resolution data synthesis method for binary image segmentation, the present invention also provides a high-resolution data synthesis device for binary image segmentation, including a memory and a processor, the memory is used to store a computer program, and the processor is used to implement the high-resolution data synthesis method for binary image segmentation when executing the computer program.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The high-resolution data synthesis method for binary image segmentation provided by the present invention generates high-quality and high-resolution synthetic data based on a two-stage method of mask editing and image generation compared to existing methods, which significantly improves the authenticity and detail fidelity of the synthetic data; wherein, mask editing uses rigid and non-rigid editing to generate diversified synthetic masks, effectively expanding the scale of training data, and topology-preserving adversarial training in non-rigid editing ensures that the synthetic mask maintains consistency of the overall structure while changing locally, thereby improving the quality of the synthetic data; image generation adopts a multi-conditional control generation method to ensure the precise correspondence between the generated high-resolution image and the synthetic mask, meeting the pixel-level accuracy requirements of the binary image segmentation task and improving the generalization ability of the model.

[0048] The present invention also provides a high-resolution data synthesis device for binary image segmentation, which is used for high-resolution data synthesis for binary image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 Schematic diagram of the process of high-resolution data synthesis method for bisection image segmentation.

[0051] Figure 2 Schematic diagram showing the effects of original data, synthetic data after rigid editing, and synthetic data after non-rigid editing.

[0052] Figure 3 It is a schematic diagram showing the original data and the effect after being processed using the high-resolution data synthesis method provided by the present invention. DETAILED DESCRIPTION

[0053] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0054] In order to synthesize high-resolution image data for the binary image segmentation task, an embodiment provides a high-resolution data synthesis method for binary image segmentation based on a two-stage method of mask editing and image generation to generate high-quality and high-resolution synthetic data, significantly improving the authenticity and detail fidelity of the synthetic data and maintaining the geometric alignment and multi-view consistency of the synthetic data.

[0055] Combine the following Figure 1 Specifically, a high-resolution data synthesis method for binary image segmentation of the present invention is introduced, which includes the following steps:

[0056] S1. Obtain an original data set, where the original data set includes an original image and a segmentation mask corresponding to the original image.

[0057] In the embodiment, the original data set is obtained in, is the original image, is the segmentation mask corresponding to the original image, where the number of original data is i~N, and N original data constitute the original data set. The original data set can be any public or private binary image segmentation data set.

[0058] S2. Perform rigid editing and non-rigid editing on the segmentation mask corresponding to the original image in the original data set to obtain an edited synthetic mask set, wherein the edited synthetic mask set includes a rigidly edited synthetic mask set and a non-rigidly edited synthetic mask set.

[0059] In the embodiment, based on the segmentation mask in the original data set D Perform rigid editing and obtain rigid editing synthesis mask set in, represents each rigid edit synthesis mask, the number of rigid edit synthesis masks is i~M, and M rigid edit synthesis masks constitute the rigid edit synthesis mask set G m (rigid).

[0060] The rigid editing method includes but is not limited to editing operations such as rotation, translation, and scaling that keep the overall structure of the mask unchanged. The specific steps include:

[0061] (1) For each segmentation mask I, randomly generate a relative camera rotation and translation T of the target view i , where the number of segmentation masks is i~m. The random generation here can use uniform distribution or Gaussian distribution to determine the rotation angle and translation distance;

[0062] (2) The segmentation mask Perform inversion processing to obtain the corresponding inverted image The inversion process can simply change the pixel value from 0 to 1 and from 1 to 0;

[0063] (3) Use the pre-trained view-conditional diffusion model ψ θ , to invert the image and pan T i As input, generate a rigid edit synthesis mask The diffusion model here can adopt an architecture similar to StableDiffusion, but it needs to introduce additional view conditions to build a pre-trained view condition diffusion model ψ θ ;

[0064] (4) Repeat the above steps until a sufficient number (M) of rigid edit synthesis masks are obtained to form the rigid edit synthesis mask set G. m Typically, multiple rigidly edited composite masks of different perspectives are generated for each segmentation mask I Then construct a rigid edit synthesis mask set, the rigid edit synthesis data is as follows Figure 2 shown.

[0065] In the embodiment, based on the segmentation mask in the original data set D Perform non-rigid editing and combine the cross-attention mechanism and topology-preserving adversarial training to construct a non-rigid editing synthesis mask set G m (non- in, represents each non-rigid edit synthesis mask, the number of non-rigid edit synthesis masks is i~M, and M non-rigid edit synthesis masks constitute the non-rigid edit synthesis mask set G m (non-rigid).

[0066] Non-rigid editing methods include but are not limited to editing operations that can change mask details, such as local deformation and topological structure changes. The specific steps include:

[0067] (1) Construct a generator network G with parameter θ θ and a discriminator network D with parameter φ φ , where the generator network G θ Using U-Net structure, discriminator network D φ Adopt PatchGAN structure;

[0068] (2) For segmentation mask Generate random noise z and corresponding text prompt p, where the random noise z is usually sampled from a Gaussian distribution and the text prompt p can be a natural language sentence describing the desired change, such as Figure 1 Tips for modifying shapes in: a round table;

[0069] (3) Using the generator network G θ , based on random Gaussian noise z, text hint p and segmentation mask Generating non-rigid edit synthesis masks

[0070] (4) Using the cross-attention mechanism, the non-rigid edits are synthesized into masks The query feature Q and segmentation mask The key-value features (K, V) of are aligned to ensure non-rigid editing of the synthesis mask With segmentation mask Keep consistent in the key area, where K represents the key feature and V represents the value feature. Then extract the synthetic mask M from the cross-attention map for subsequent loss calculation to guide the model's focus.

[0071] (5) Optimize the generator network Gθ and the discriminator network D through topology-preserving adversarial training φ , where the specific steps of topology-preserving adversarial training include:

[0072] (1) Segmentation mask and non-rigid edit synthesis mask Apply edge detection operator E respectively to extract the key point set of topological structure where v j Represents each key point, the number of key points ranges from j to N v , N v Key points constitute the key point set V; here the edge detection operator E can use the Canny edge detector or the Sobel operator;

[0073] (2) Based on the key point set V and the edge detection operator E, a topological structure graph T = (V, E) is constructed, and the connection relationship of the edges is determined using the k-nearest neighbor algorithm;

[0074] (3) Using the discriminator network D φ For the segmentation mask topology graph T s and non-rigid editing synthesis mask topology map T g Conduct adversarial training to maintain the consistency of the topological structure, where the discriminator network D φ Parameterize to maximize the training objective:

[0075]

[0076] Where T s represents the segmentation mask topology, p represents the text prompt, T s ~p data (T s ) represents the segmentation mask topology graph based on the text hint p, T g represents the non-rigid edit synthesis mask topology graph, T g ~p gen (T g ) represents the non-rigid edit synthesis mask topology graph based on the text hint p;

[0077] For the generator network G θ Parameterization to minimize the training objective:

[0078]

[0079] Where T g represents the non-rigid edit synthesis mask topology graph, T g ~p gen (T g ) represents the topological structure of the non-rigid edit synthesis mask based on the text prompt p. Based on the optimization, the non-rigid edit synthesis mask set is constructed. The non-rigid edit synthesis data is as follows Figure 2 shown.

[0080] S3. Based on the edited synthetic mask set, a multi-conditional control generation method is used to generate the corresponding high-resolution image.

[0081] In the embodiment, based on the edited synthesis mask set G m , a multi-condition control generation method is used to generate the corresponding high-resolution image set in Represents the high-resolution image corresponding to the edited synthetic mask. The number of high-resolution images is i~M, and M high-resolution images constitute the high-resolution image set G r .

[0082] The multi-conditional control generation method uses the edited synthetic mask set as the segmentation condition, the edge information as the structural condition and the text description as the semantic condition. The specific steps include:

[0083] (1) Edit the composite mask As a split condition in Includes rigid edit compositing masks and non-rigid edit synthesis mask Split Condition The number of is i~s;

[0084] (2) Edited composite mask Apply the Canny operator to get the edge condition The edge conditions The number of is i~y, and the parameters of the Canny operator need to be adjusted according to the specific task to obtain the best edge information;

[0085] (3) The split condition and edge conditions Input to control block B θ , control block B θ It can be a lightweight convolutional neural network used to process and fuse different conditional information;

[0086] (4) Control block B θ The output is injected into the pre-trained diffusion model M with parameters θ θ In the process of controlling the denoising of random Gaussian noise z, a corresponding high-resolution image is generated.

[0087]

[0088] Among them, the diffusion model M θ A similar architecture to Stable Diffusion can be used, but it needs to adapt to the input of multi-conditional control; repeat the above steps until all edited synthetic masks are obtained to generate corresponding high-resolution images, and construct a high-resolution image set G r .

[0089] S4. Combining the edited synthetic mask set with the corresponding high-resolution image to obtain a high-resolution synthetic dataset for binary image segmentation.

[0090] In the embodiment, the edited composite mask set G m and the corresponding high-resolution image set G r Combined to obtain the final high-resolution synthetic dataset in, represents the final high-resolution synthetic data, the number of high-resolution synthetic data is i~M, and M high-resolution synthetic data constitute the high-resolution synthetic data set G.

[0091] Figure 3 The effects of some high-resolution data synthesis methods described above for bisection image segmentation are demonstrated.

[0092] An embodiment also provides a high-resolution data synthesis device for binary image segmentation, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to implement the high-resolution data synthesis method for binary image segmentation when executing the computer program.

[0093] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A high-resolution data synthesis method for binary image segmentation, characterized in that: The following steps are involved: Obtain an original data set, where the original data set includes an original image and a segmentation mask corresponding to the original image; Performing rigid editing and non-rigid editing on the segmentation mask corresponding to the original image in the original data set to obtain an edited synthetic mask set, wherein the edited synthetic mask set includes a rigidly edited synthetic mask set and a non-rigidly edited synthetic mask set; Based on the edited synthetic mask set, a multi-conditional control generation method is used to generate the corresponding high-resolution image; The edited synthetic mask set and the corresponding high-resolution image are combined to obtain a high-resolution synthetic dataset for bisection image segmentation.

2. The high-resolution data synthesis method for binary image segmentation according to claim 1, characterized in that: The rigid editing adopts one or more of the editing operations of rotation, translation and scaling.

3. The high-resolution data synthesis method for binary image segmentation according to claim 2, characterized in that: The rigid editing steps include: (1) For each segmentation mask I, randomly generate a relative camera rotation and translation T of the target view i , where the number of segmentation masks is i~m; (2) The segmentation mask Perform inversion processing to obtain an inverted image (3) Use the pre-trained view-conditional diffusion model ψ θ , to invert the image and pan T i As input, a rigid edit synthesis mask is generated; (4) Repeat steps (1) to (3) to obtain multiple rigid edit synthesis masks and construct a rigid edit synthesis mask set.

4. The high-resolution data synthesis method for binary image segmentation according to claim 1, characterized in that: The non-rigid editing adopts one or more of local deformation, topology, and scaling editing operations.

5. The high-resolution data synthesis method for binary image segmentation according to claim 4, characterized in that: The non-rigid editing steps include: (1) Construct a generator network G with parameter θ θ and a discriminator network D with parameter φ φ ; (2) For segmentation mask Generate random Gaussian noise z and corresponding text prompt p; (3) Using the generator network G θ , based on random Gaussian noise z, text hint p and segmentation mask Generate non-rigid edit synthesis mask; (4) Using the cross-attention mechanism, the query feature Q of the non-rigid edit synthesis mask is combined with the segmentation mask The key-value features (K, V) of the proposed algorithm are aligned, where K represents the key feature and V represents the value feature. A synthetic mask M is extracted from the cross-attention map to guide the focus of the model. (5) Optimize the generator network G through topology-preserving adversarial training θ and the discriminator network D φ ,Construct a set of non-rigid edit synthesis masks.

6. The high-resolution data synthesis method for binary image segmentation according to claim 5, characterized in that: The topology preserving adversarial training includes: (1) Segmentation mask The edge detection operator E is applied to the non-rigid edit synthesis mask to extract the key point set of the topological structure. where v j Represents each key point, the number of key points ranges from j to N v ; (2) Based on the key point set V and the edge detection operator E, a topological structure graph T = (V, E) is constructed; (3) Using the discriminator network D φ For the segmentation mask topology graph T s And the composite mask topology structure diagram T g Conduct adversarial training.

7. The high-resolution data synthesis method for binary image segmentation according to claim 5, characterized in that: The optimized generator network G θ and the discriminator network D φ include: For the discriminator network D φ Training to the maximum: Where T s represents the segmentation mask topology, p represents the text prompt, T s ~p data (T s ) represents the segmentation mask topology diagram based on the text hint p, T g represents the non-rigid edit synthesis mask topology graph, T g ~p gen (T g ) represents the non-rigid edit synthesis mask topology graph based on the text hint p; For the generator network G θ Training is minimized: Where T g represents the non-rigid edit synthesis mask topology graph, T g ~p gen (T g ) represents the non-rigid edit synthesis mask topology graph based on the text hint p.

8. The high-resolution data synthesis method for binary image segmentation according to claim 1, characterized in that: The claimed multi-conditional controlled generation method adopts the edited synthetic mask set as the segmentation condition, the edge information as the structural condition and the text description as the semantic condition.

9. The high-resolution data synthesis method for binary image segmentation according to claim 8, characterized in that: The multi-condition control generation method comprises the following steps: (1) Edit the composite mask As a split condition The number of edited composite masks is i~m, and the number of segmentation conditions is i~s; (2) Edited composite mask Apply edge detection operator to get edge condition The number of edge conditions is i~y; (3) The split condition and edge conditions Input to control block B θ ; (4) Control block B θ The output is injected into the pre-trained diffusion model M with parameters θ θ In the process of controlling the denoising of random Gaussian noise z, a corresponding high-resolution image is generated.

10. A high-resolution data synthesis device for binary image segmentation, comprising a memory and a processor, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the high-resolution data synthesis method for binary image segmentation according to any one of claims 1 to 9 when executing the computer program.

Images