A method for image generation containing segmentation masks

Abstract

The present invention discloses a method for generating an image containing a segmentation mask, comprising: obtaining an unlabeled training image; using an image editing algorithm to obtain target category information in the unlabeled training image; obtaining a segmentation mask of the training image based on the target category information and the image editing algorithm; adjusting the brightness of the training image based on the segmentation mask to obtain a training image containing a segmentation mask; training a Vincent graph model using the training image containing the segmentation mask; based on the Vincent graph model, inputting any target category name to be generated, and generating an image containing the target category and containing a segmentation mask. The present invention can automatically obtain a segmentation mask of an image while generating an image; it can generate images not only for categories that have appeared on the training image, but also for categories that have not appeared on the training image, that is, it has a Vincent graph capability for open vocabulary categories.

Description

Technical Field

[0001] The invention belongs to the field of image generation, and in particular relates to an image generation method containing a segmentation mask. Background Art

[0002] The text-guided image generation model, i.e., the text-generated image model, has a wide range of applications. However, existing methods only generate images. If you want to understand the generated images, you often need to design additional algorithms to obtain the segmentation mask of the generated images, i.e., to determine whether each pixel in the generated image is a target or background. In order to achieve image generation with segmentation masks, it is often necessary to manually annotate each training image at the pixel level. In view of the fact that manual annotation requires a lot of manpower and material resources, there is an urgent need for a method to generate images with segmentation masks based on unlabeled training images. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes an image generation method containing a segmentation mask. While generating an image, the present invention can also automatically obtain the segmentation mask of the image; it can not only generate images for categories that have appeared in the training images, but also generate images for categories that have not appeared in the training images, that is, it has the ability of text-based graphs for open vocabulary categories.

[0004] To achieve the above object, the present invention provides an image generation method containing a segmentation mask, comprising:

[0005] Obtain unlabeled training images;

[0006] Using an image editing algorithm, obtaining target category information in the unlabeled training image;

[0007] Based on the target category information and the image editing algorithm, obtaining a segmentation mask of the training image;

[0008] Adjusting the brightness of the training image based on the segmentation mask to obtain a training image containing the segmentation mask;

[0009] A Vincent graph model is trained using a training image containing a segmentation mask, and an image containing a segmentation mask is obtained based on the Vincent graph model.

[0010] Optionally, using an image editing algorithm to obtain target category information in the unlabeled training image includes:

[0011] Get a collection containing target categories;

[0012] Based on each target category in the set, designing an editing prompt language containing the category;

[0013] Input the editing prompt language as a control condition into the image editing algorithm, edit the unlabeled training image, and obtain the edited image containing the category;

[0014] Compare the color difference between the edited image containing the category and the unlabeled training image, and obtain the category number with the smallest color difference;

[0015] Based on the category number, target category information is determined.

[0016] Optionally, based on the category number, determining the target category information includes:

[0017] If the color difference value corresponding to the category number is less than a preset threshold, obtaining target category information;

[0018] Otherwise, the unlabeled training image is deleted from the training image dataset.

[0019] Optionally, based on the target category information and the image editing algorithm, obtaining a segmentation mask of the training image includes:

[0020] Designing a pair of differentiated editing prompt languages ​​using training images containing the target category information;

[0021] Inputting the editing prompt language as a control condition into an image editing algorithm to obtain a pair of color difference images;

[0022] The color difference of the color difference image is calculated, and binarization and morphological filtering are performed to obtain the segmentation mask.

[0023] Optionally, the method for adjusting the brightness of the training image based on the segmentation mask is:

[0024]

[0025] Among them, V and U represent the real brightness and the brightness containing the segmentation mask respectively, α is the adjustment coefficient, i is the pixel, and M(i) is the mask corresponding to the pixel.

[0026] Optionally, based on the Wensheng graph model, obtaining an image containing a segmentation mask includes:

[0027] Inputting the name of any target category to be generated into the text graph model to obtain an image containing an initial implicit segmentation mask of the category;

[0028] According to the brightness value, the brightness of the image containing the initial segmentation mask is restored, and the image containing the segmentation mask is output.

[0029] Optionally, the method for restoring the brightness of the image initially containing the segmentation mask according to the brightness value includes:

[0030]

[0031] Among them, V and U represent the real brightness and the brightness containing the segmentation mask respectively, α is the adjustment coefficient, i is the pixel, and M(i) is the mask corresponding to the pixel.

[0032] Compared with the prior art, the present invention has the following advantages and technical effects:

[0033] The present invention proposes an image generation method containing segmentation masks. Based on the image editing method, the method can generate category information and segmentation masks as pseudo-annotations for a large number of unlabeled training images, and can contain the segmentation mask information into the training images by adjusting the image brightness. The method can generate images and segmentation masks simultaneously by fine-tuning the training text graph model, and can generate images for categories that have not appeared in the training images, that is, it has the text graph capability for open vocabulary categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0035] Figure 1 is a flow chart of an image generation method including a segmentation mask according to the present embodiment;

[0036] Figure 2 This is an example diagram of obtaining target categories in an image based on image editing in this embodiment;

[0037] Figure 3 This is an example diagram of obtaining a segmentation mask based on image editing in this embodiment, where (a) is a training image, (b) and (c) are edited images, and (d) is a target segmentation mask diagram;

[0038] Figure 4 1 is an example diagram of embedding segmentation mask information into an image through brightness adjustment in this embodiment, wherein (a) is a training image, (b) is a segmentation mask obtained based on image editing, and (c) is an image embedding the segmentation mask;

[0039] Figure 5 This is a diagram of extracting a segmentation mask and an image example from an image generated by the text-generated graph model SD_FT in this embodiment. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] The images in this embodiment include different categories, mainly images of people, images of dogs and images of horses, which are represented by "person", "horse" and "dog" respectively.

[0043] This embodiment provides a method for generating an image containing a segmentation mask, such as Figure 1 As shown, the specific steps include:

[0044] Obtain unlabeled training images;

[0045] Use image editing algorithms to obtain target category information in unlabeled training images;

[0046] Based on the target category information and the image editing algorithm, the segmentation mask of the training image is obtained;

[0047] Adjusting the brightness of the training image based on the segmentation mask to obtain a training image containing the segmentation mask;

[0048] The training image containing the segmentation mask is used to train the Vincent graph model, and the image containing the segmentation mask is obtained based on the Vincent graph model.

[0049] Specifically, this embodiment proposes a method for generating an image containing a segmentation mask. First, based on an unlabeled training image dataset, a category label is generated for the training image by an image editing method, and then a segmentation mask is generated for the training image as a pseudo-label by an image editing method, thereby realizing an efficient method that does not require manual labeling of data. Secondly, the segmentation mask information is embedded in the training image by an image brightness adjustment method, and the image generation containing the segmentation mask is realized by fine-tuning the existing image generation model.

[0050] Furthermore, the image editing algorithm is used to obtain the target category information in the unlabeled training image, including:

[0051] Get a collection containing target categories;

[0052] Based on each target category in the set, design an editing prompt language containing the category;

[0053] Input the editing prompt language as a control condition into the image editing algorithm, edit the unlabeled training image, and obtain the edited image containing the category;

[0054] Compare the color difference between the edited image containing the category and the unlabeled training image, and obtain the category number with the smallest color difference;

[0055] Based on the category number, the target category information is determined.

[0056] Further, based on the category number, determining the target category information includes:

[0057] If the color difference value corresponding to the category number is less than the preset threshold, the target category information is obtained;

[0058] Otherwise, the unlabeled training image is deleted from the training image dataset.

[0059] Specifically, in order to avoid the manpower and material resources required for manually labeling a large number of images, this method is based on an unlabeled training image dataset. First, the category information of the target in each training image is obtained through an image editing method. Given a set containing K target categories The design contains each category C i The editing prompt language of "aphoto ofC i ”, as a control condition input to the text-guided image editing algorithm, edit the unlabeled training image I, and generate i Image I i , calculate and compare the image I generated by editing i The color difference between the training image I and the image I is used to obtain the category number m with the smallest color difference, as shown in formula (1):

[0060]

[0061] Where |II i | represents image I and image I i The average of the absolute values ​​of all pixel color differences. If the color difference value corresponding to category number m |II m | is less than a certain threshold, then category C m The category information CLS of the training image I is determined to be, otherwise the acquisition of the category information fails, and the image I is deleted from the training image data set.

[0062] Figure 2 A specific example is shown, in which (a) is a training image I, (b) and (d) are images I1 and I2 obtained after editing by the image editing algorithm based on the editing prompt language "a photo of horse" and "a photo of dog", (c) and (e) are the color differences between the edited images I1 and I2 and image I, respectively. The category with the smaller color difference is "horse", which is the category CLS of the training image I.

[0063] Furthermore, based on the target category information and the image editing algorithm, obtaining the segmentation mask of the training image includes:

[0064] Design a pair of editing prompts with obvious differences for the training images containing target category information;

[0065] Input the editing prompt language as a control condition into the image editing algorithm, edit the training image, and obtain a pair of color difference images;

[0066] Calculate the color difference of the color difference image, perform binarization and morphological filtering, and obtain the segmentation mask.

[0067] Specifically, for a training image I known to contain the target category CLS, a pair of editing hints containing category information CLS but with obvious differences are designed, such as "a photo of black CLS" and "a photo of white CLS", and are input into the text-guided image editing algorithm as control conditions to edit the training image I into two images I1 and I2 with consistent structure but obvious color differences. Since the hints used to guide image editing only modify the target, the two edited images I1 and I2 only have obvious changes in the target area, and the background area changes very little. By calculating the color difference between the two edited images I1 and I2, and performing binarization and morphological filtering, the target segmentation mask M is obtained.

[0068] Figure 3 A specific example is shown, where Figure 3 (a) is the training picture, Figure 3 (b) and Figure 3 (c) is the image edited by the image editing algorithm based on the editing prompt language "a photo of black horse" and "a photo of white horse". By calculating the color difference between the two edited images and performing binarization and morphological filtering, the target segmentation mask is obtained as shown in Figure 3 (d) as shown.

[0069] Furthermore, the brightness of the training image is adjusted based on the segmentation mask to obtain a training image containing the segmentation mask.

[0070] For a training image I represented in HSV color space HSV , where hue H, saturation S and brightness V range from 0 to 1. In order to retain the color information in the image and contain the information of the segmentation mask M, this method adjusts the brightness V of each pixel in the training image to obtain the training image I containing the segmentation mask HSU , as shown in formula (2):

[0071]

[0072] Where V and U represent the real brightness and the brightness implied by the segmentation mask, respectively, and α is the adjustment coefficient. For each pixel i, if the pixel belongs to the background, that is, M(i) = 0, the V value will be multiplied by the coefficient α, that is, it will be adjusted to a lower range (0 to α); if the pixel belongs to the target, that is, M(i) = 1, the V value will be adjusted to a higher range (1-α to 1). α is limited to < 0.5 to ensure that the adjusted value ranges of the target and background pixels do not overlap.

[0073] Figure 4 Shows an example of embedding segmentation mask information into training images by adjusting pixel brightness. Figure 4 (a) is the training image I HSV , Figure 4 (b) is the segmentation mask M obtained based on image editing. Figure 4 (c) is the image I containing the segmentation mask HSU , the target in the image will be highlighted and the background will be relatively darkened, thus embedding the segmentation mask information into the image.

[0074] Furthermore, based on the training images containing segmentation masks, fine-tuning training is performed to obtain the Vincent graph model SD_FT.

[0075] The existing text-based graph model SD can generate images of various target categories and has good category generalization, but it cannot automatically generate segmentation masks. In order to make full use of the category generalization ability of the existing text-based graph model and generate images containing segmentation masks, this method is based on training images containing segmentation masks. On the basis of the existing text-based graph model SD, a large model fine-tuning training method is used to enable the model to generate images containing segmentation masks.

[0076] In order to guide the Wensheng graph model to learn the correspondence between target categories, brightness adjustment instructions and image content, this method is used for training images I containing target categories CLS and brightness adjusted HSU , a prompt language containing category information and brightness adjustment commands is designed: "a photo of CLS, object highlight, background darken", where CLS indicates that the image contains the category information of the target, and "object highlight, background darken" indicates the brightness adjustment instruction. Fine-tuning training is performed on the existing text image model SD to obtain the fine-tuned model SD_FT, which can generate images containing segmentation masks.

[0077] Furthermore, based on the Wensheng graph model, obtaining an image containing a segmentation mask includes:

[0078] Input the name of any target category to be generated into the Wensheng graph model to obtain an image containing the initial implicit segmentation mask of the category;

[0079] According to the brightness value, the brightness of the image containing the initial segmentation mask is restored, and the image containing the segmentation mask is output.

[0080] Specifically, the prompt language "aphoto of CLS, object highlight, background darken" containing arbitrary category information CLS and brightness adjustment command is input into the fine-tuned trained text image model SD_FT to generate an image I containing the category CLS and the segmentation mask. HSU , according to the value of the brightness U containing the segmentation mask, it is determined whether each pixel is a target and the true brightness V value is restored, as shown in formula (3).

[0081]

[0082] Among them, U and V represent the brightness of the segmentation mask and the real brightness respectively, and the M(i) value indicates whether each pixel i is the target pixel. Through formula (3), the segmentation mask M and the generated image I represented in the HSV color space are obtained at the same time HSV .

[0083] Although only a set of K target categories is used when fine-tuning the Wensheng graph model However, the fine-tuned model can generate images containing targets of other categories beyond these K categories, that is, it has the ability to generate open vocabulary-oriented text images.

[0084] Figure 5 The first row is an image generated by the Vincent graph model SD_FT with a segmentation mask, the second row is the segmentation mask extracted from the generated image, and the third row is the result of restoring the real image from the generated image. Figure 5 The first three columns belong to the K target categories used in fine-tuning training. The category "Backpack" in the fourth column does not belong to the K target categories used in fine-tuning training, but it is still possible to generate pictures containing this category, extract segmentation masks and restore real images, demonstrating the ability of text-based graphs for open vocabulary.

[0085] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for generating an image containing a segmentation mask, characterized in that: include: Obtain unlabeled training images; Using an image editing algorithm, obtaining target category information in the unlabeled training image; Using an image editing algorithm to obtain target category information in the unlabeled training image includes: Get a collection containing target categories; Based on each target category in the set, designing an editing prompt language containing the category; Input the editing prompt language as a control condition into the image editing algorithm, edit the unlabeled training image, and obtain the edited image containing the category; Compare the color difference between the edited image containing the category and the unlabeled training image, and obtain the category number with the smallest color difference; Based on the category number, determining target category information; Based on the category number, determining the target category information includes: If the color difference value corresponding to the category number is less than a preset threshold, the target category information is obtained, wherein the difference value is the average of the absolute values ​​of the color differences of all pixels between the unlabeled training image and the edited image containing the category; Otherwise, the unlabeled training image is deleted; Based on the target category information and the image editing algorithm, obtaining a segmentation mask of the training image; Based on the target category information and the image editing algorithm, obtaining a segmentation mask of the training image includes: Designing a pair of editing prompt languages ​​with obvious differences for the training images containing the target category information; Input the editing prompt language as a control condition into the image editing algorithm, edit the training image, and obtain a pair of color difference images; Calculating the color difference of the color difference image, and performing binarization and morphological filtering to obtain the segmentation mask; Adjusting the brightness of the training image based on the segmentation mask to obtain a training image containing the segmentation mask; A Vincent graph model is trained using a training image containing a segmentation mask, and an image containing a segmentation mask is obtained based on the Vincent graph model.

2. The method for generating an image containing a segmentation mask according to claim 1, characterized in that: The method for adjusting the brightness of the training image based on the segmentation mask is: Among them, V and U represent the real brightness and the brightness of the implicit segmentation mask respectively, α is the adjustment coefficient, i is the pixel, and M(i) is the mask corresponding to the pixel.

3. The method for generating an image containing a segmentation mask according to claim 1, characterized in that: Based on the Wensheng graph model, obtaining an image containing a segmentation mask includes: Inputting the name of any target category to be generated into the text graph model to obtain an image containing an initial implicit segmentation mask of the category; According to the brightness value, the brightness of the image containing the initial segmentation mask is restored, and the image containing the segmentation mask is output.

4. The method for generating an image containing a segmentation mask according to claim 3, characterized in that: The method of restoring the brightness of the image initially containing the segmentation mask according to the brightness value comprises: Among them, V and U represent the real brightness and the brightness containing the segmentation mask respectively, α is the adjustment coefficient, i is the pixel, and M(i) is the mask corresponding to the pixel.

Images