An open domain segmentation method based on counterfactual image generation using diffusion model

Abstract

The present invention relates to the technical field of computer vision and proposes an open domain segmentation method for counterfactual image generation based on a diffusion model. The method comprises the following steps: obtaining a vocabulary consisting of several categories of words; generating a text prompt using a large language model based on the vocabulary; creating a counterfactual text based on the text prompt; generating an original image using a diffusion model based on the text prompt; generating a counterfactual image using a diffusion model based on the counterfactual text; extracting a true mask from the original image using an open world detection model and an interactive segmentation model; obtaining an open world segmentation model for predicting an image mask; and optimizing and training the open world segmentation model using the true mask and the counterfactual image. When the ability of the open world segmentation model to predict the image mask reaches an optimal level or the number of training times reaches a preset value, training is stopped to obtain an optimized open world segmentation model.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and more particularly to an open domain segmentation method based on diffusion model counterfactual image generation. Background Art

[0002] The emergence of visual language models has given rise to open-world segmentation models, which exploit the correlation between images and text for segmentation. These models draw their segmentation power from a vast library of image-text pairs. However, due to a lack of pixel-level annotated data, their segmentation capabilities are relatively limited compared to closed-set semantic segmentation paradigms. To enhance the segmentation and localization capabilities of open-world segmentation models, a dataset with diverse categories and pixel-level annotations is essential, and creating such a dataset requires significant investment in human resources.

[0003] Existing technologies create datasets for training open-world segmentation models by synthesizing image-text pair data and automatically obtaining pixel-level annotations for the synthesized data; DatasetDM uses an additional decoder trained on a small amount of data to annotate the generated data, while FreeMask enhances training by generating images from existing pixel labels; existing technologies still need to obtain a small amount of real data with segmentation labels, which is difficult to achieve for the construction of open-domain segmentation datasets; in addition, existing technologies mainly focus on building datasets for specific closed sets, and there is currently no method for creating segmentation datasets suitable for open-world scenarios.

[0004] In summary, the existing technology has the disadvantage of being difficult to obtain real data when creating segmentation datasets for open world scenes. Summary of the Invention

[0005] In order to overcome the defect of the above-mentioned prior art that it is difficult to obtain real data, the present invention provides an open domain segmentation method based on diffusion model counterfactual image generation without the need for real data.

[0006] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0007] Obtain a vocabulary consisting of several categories of words;

[0008] generating text prompts based on the vocabulary using a large language model;

[0009] Create counterfactual text based on text prompts;

[0010] Based on the text prompt, the original image is generated using the diffusion model;

[0011] Based on counterfactual text, a diffusion model is used to generate counterfactual images;

[0012] Leveraging open-world detection models and interactive segmentation models to extract ground-truth masks from raw images;

[0013] Get an open-world segmentation model for predicting image masks;

[0014] The open-world segmentation model is optimized and trained using real masks and counterfactual images. When the open-world segmentation model's ability to predict image masks reaches an optimal level or the number of training times reaches a preset value, the training is stopped to obtain an optimized open-world segmentation model.

[0015] The present invention also proposes an open domain segmentation system based on diffusion model counterfactual image generation for implementing the above-mentioned open domain segmentation method based on diffusion model counterfactual image generation. The system includes:

[0016] A vocabulary acquisition module is used to acquire a vocabulary consisting of several categories of words;

[0017] a text prompt generation module, configured to generate text prompts based on the vocabulary using a large language model;

[0018] A counterfactual text generation module for creating counterfactual text based on text prompts;

[0019] An original image generation module, used to generate an original image based on a text prompt using a diffusion model;

[0020] A counterfactual image generation module, which is used to generate counterfactual images based on counterfactual text using a diffusion model;

[0021] A mask extraction module, which extracts ground-truth masks from raw images using an open-world detection model and an interactive segmentation model;

[0022] A model acquisition module for acquiring an open-world segmentation model for predicting image masks;

[0023] A model training module is used to optimize the training of the open-world segmentation model using real masks and counterfactual images. When the open-world segmentation model's ability to predict image masks reaches an optimal level or the number of training times reaches a preset value, the training is stopped to obtain an optimized open-world segmentation model.

[0024] The present invention also proposes a computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor performs the steps of the above-mentioned open domain segmentation method based on diffusion model counterfactual image generation.

[0025] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0026] A text prompt is generated based on the vocabulary, and the original image is generated using the diffusion model. The open-world detection model and the interactive segmentation model are used to extract the real mask from the original image, thereby obtaining synthetic data and pixel-level annotations of the synthetic data without the need for real data. At the same time, counterfactual text is created based on the text prompt, and the diffusion model is used to generate counterfactual images. The counterfactual images are used to optimize the training of the open-world segmentation model, reduce the noise of the pixel-level annotations, and enhance the segmentation and positioning capabilities of the open-world segmentation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of the open domain segmentation method based on diffusion model counterfactual image generation proposed in Example 1;

[0028] Figure 2 This is a schematic diagram of the framework of the open domain segmentation method proposed in Example 1;

[0029] Figure 3 Schematic diagram of the counterfactual image pair proposed in Example 2;

[0030] Figure 4 This is a visual comparison diagram proposed in Example 2;

[0031] Figure 5 This is an example diagram of the visualization of the open world segmentation results proposed in Example 2;

[0032] Figure 6 This is the overall framework diagram of an open domain segmentation system based on diffusion model counterfactual image generation proposed in Example 3. DETAILED DESCRIPTION

[0033] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting the present embodiment;

[0034] In order to better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product size;

[0035] It is understandable to those skilled in the art that some well-known structures and descriptions thereof may be omitted in the drawings.

[0036] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0037] Example 1

[0038] This embodiment proposes an open domain segmentation method based on counterfactual image generation based on a diffusion model. Figure 1Schematic diagram of the process of the open domain segmentation method based on diffusion model counterfactual image generation of this embodiment;

[0039] The open domain segmentation method proposed in this embodiment based on counterfactual image generation based on a diffusion model includes the following steps:

[0040] S1: Obtain a vocabulary consisting of several categories of words;

[0041] S2: Generate text prompts based on the vocabulary using a large language model;

[0042] S3: Create counterfactual text based on text prompts;

[0043] S4: Generate the original image using the diffusion model based on the text prompt;

[0044] S5: Generate counterfactual images based on counterfactual text using diffusion model;

[0045] S6: Extracting ground-truth masks from original images using an open-world detection model and an interactive segmentation model;

[0046] S7: Obtain an open-world segmentation model for predicting image masks;

[0047] S8: Optimize and train the open-world segmentation model using real masks and counterfactual images. When the open-world segmentation model's ability to predict image masks reaches an optimal level, or the number of training times reaches a preset value, stop training to obtain an optimized open-world segmentation model.

[0048] In the specific implementation process, text prompts are generated based on the vocabulary, the diffusion model is used to generate the original image, and the open-world detection model and the interactive segmentation model are used to extract the real mask from the original image, so as to obtain synthetic data and pixel-level annotations of the synthetic data without the need for real data. At the same time, counterfactual text is created based on the text prompt, and the diffusion model is used to generate counterfactual images. The open-world segmentation model is optimized and trained using the counterfactual images to reduce the noise of the pixel-level annotations and enhance the segmentation and positioning capabilities of the open-world segmentation model.

[0049] In an optional embodiment, the vocabulary is expressed as:

[0050] V={c1,...,c N}

[0051] In the formula, c1 represents the first category of vocabulary, c N Represents the Nth category of vocabulary, where N represents the total number of categories of vocabulary;

[0052] The steps of generating a text prompt using a large language model based on the vocabulary include:

[0053] Extract n categories of vocabulary from the vocabulary, input the class names and conditional statements of the extracted n categories of vocabulary into the large language model, and the large language model outputs a text prompt; where the size of n is a preset value, and n < N, and the conditional statements include: "The scenarios and details of these examples are as diverse as possible" and / or "The features of the object are described as detailed as possible."

[0054] As an illustrative example, each category of vocabulary corresponds to several text prompts.

[0055] In this optional embodiment, in order to ensure that there are various different categories in the dataset, the present application establishes a vocabulary V containing a large number of categories, and the size of the vocabulary is N; high-quality descriptions are of great importance for generating images with complex details, so there are certain constraints on the number of categories n when generating a text; as an illustrative example, one or two class names are selectively extracted from the vocabulary to create a text, and in order to guide the large language model to generate the corresponding text, a set of conditions are called, including "The scenarios and details of these examples are as diverse as possible", "The features of the object are described as detailed as possible", and as an illustrative example, the large language model includes: ChatGPT.

[0056] In an optional embodiment, the steps of creating a counterfactual text based on the text prompt include:

[0057] Replace the class name of the vocabulary in the text prompt with a replacement statement to form a counterfactual text;

[0058] Among them, the replacement statement includes: "nothing".

[0059] As an illustrative example, each text prompt corresponds to a counterfactual text.

[0060] Figure 2 This is a schematic diagram of the framework of the open-domain segmentation method proposed in this embodiment. Figure 2 The left half is a schematic diagram of the framework of generation based on counterfactual diffusion, as Figure 2 shown in the left half. In this optional embodiment, the class name "dog" is extracted from the vocabulary, and "dog" and the conditional statement are input into the large language model ChatGPT, and ChatGPT outputs the text prompt: "In a picturesque park, a naughty brown and white dog is happily chasing a bright yellow tennis ball"; when creating a counterfactual text based on the text prompt, replace "dog" in the text prompt with "nothing" to obtain the counterfactual text.

[0061] In an optional embodiment, the step of extracting a ground truth mask from an original image using an open-world detection model and an interactive segmentation model includes:

[0062] The class names of the words in the original image and the text prompt are input into the open-world detection model. The open-world detection model recognizes the objects corresponding to the class names of the words from the original image and outputs the recognition bounding box.

[0063] The interactive segmentation model extracts a ground-truth mask from the original image based on the identified bounding box;

[0064] The expression of the real mask is:

[0065]

[0066] Where Y represents the true mask, and Represents the i-th category word c in the vocabulary i and the nth category vocabulary c n The corresponding real mask, Represents a tensor with a first dimension of n, a second dimension of H, and a third dimension of W, where H represents the height of the image corresponding to the true mask, and W represents the width of the image corresponding to the true mask.

[0067] As an exemplary explanation, after creating a text prompt, this application inputs the text prompt into the diffusion model, and the diffusion model outputs the original image. Due to the high quality of the text, different images of each class can be obtained; at the same time, in order to enhance the target positioning ability of the open world model and overcome the noise in the generated image mask, the class name in the text is replaced with "nothing" to create a counterfactual text; the diffusion model is used to generate a counterfactual image corresponding to the counterfactual text. The counterfactual image obtained by class name replacement is similar to the original image, but lacks the corresponding category, such as Figure 2 As shown in the left half, the original image is a dog running on the grass, and the counterfactual image is an image of the grass without the dog. As an example, the diffusion model includes: Stable Diffusion; Figure 2 As shown on the left, the open-world detection model and the interactive segmentation model are combined to extract the true mask from the original image. As an example, the open-world detection model includes Grounding DINO, and the interactive segmentation model includes SAM (Segment Anything Model).

[0068] In this optional embodiment, for the generated original image, the open world detection model is used to identify the objects in the image based on the class names used during generation. Images that do not simultaneously generate the target object can be excluded. The resulting bounding box is then used as a hint for the interactive segmentation model to extract the mask. This enables pixel-level annotation and obtains a labeled dataset suitable for open-domain semantic segmentation, enabling the open-world segmentation model to obtain better semantic segmentation capabilities in cross-modal learning, overcoming the shortcomings of previous methods in lacking fine-grained segmentation capabilities;

[0069] In this optional embodiment, a dataset suitable for open-world segmentation is constructed, which not only has a wide variety of categories including pixel-level masks, but also contains text descriptions and counterfactual images for noise mitigation.

[0070] In an optional embodiment, the step of optimizing and training the open-world segmentation model using the real mask and the counterfactual image includes:

[0071] Extract m categories of words from the vocabulary to create a vocabulary subset C sub , subset the vocabulary C sub The original image is input into the open-world segmentation model, and the open-world segmentation model outputs the predicted mask;

[0072] Design the first predicted mask loss function based on the real mask and the predicted mask and the second predicted mask loss function

[0073] Design a comparative training loss function based on counterfactual images and original images

[0074] Based on the loss function Loss Function and loss function Designing the total loss function Iteratively solve the total loss function Until the total loss function When the minimum value is reached or the number of iterations reaches the preset number, the iteration is stopped, the optimization of the open world segmentation model is completed, and the optimized open world segmentation model is obtained;

[0075] Among them, the total loss function The expression is:

[0076]

[0077] In an optional embodiment, the first prediction mask loss function The expression is:

[0078]

[0079] In the formula, α represents the hyperparameter, represents the category c at image pixel k i The corresponding real mask; Indicates category c at pixel k i The corresponding prediction mask.

[0080] In an optional embodiment, the second prediction mask loss function The expression is:

[0081]

[0082] Where, represents the category c at image pixel k i The corresponding real mask; Indicates category c at pixel k i The corresponding prediction mask.

[0083] In an optional embodiment, the contrastive training loss function The expression is:

[0084]

[0085] Where p co Represents the class symbol corresponding to the counterfactual image, p clx represents the class symbol of the original image corresponding to the counterfactual image, and ∥·∥ represents the modulus operation of the orientation quantity.

[0086] In this optional embodiment, Figure 2 The right half is a schematic diagram of the framework for training the open world segmentation model. Figure 2 The two small boxes pointed by the arrows in the right half of the counterfactual contrastive learning both represent CLS symbols, such as Figure 2 As shown in the right half, an open-world image segmentation model (Open-World Image Segmentation Model) pre-trained based on image-text comparison is used to train the segmentation dataset. As an exemplary illustration, the open-world segmentation model includes: ZegCLIP (Towards AdaptingCLIP for Zero-shot Semantic Segmentation), which predicts masks by leveraging the correlation between image features from the image encoder and text embedding from the open-world segmentation model; however, applying a dataset with a large number of categories to open-world segmentation is a challenge; the number of categories with masks for each image is limited, and most categories in the vocabulary do not appear in the image; during the training phase, if the masks of all categories in the vocabulary of an image are predicted, that is, Due to the existence of a large number of non-existent categories (n << N), the learning efficiency of the existing categories in the image will be reduced; at the same time, this method may lead to a large computational burden and introduce more noise, thus affecting the performance; to solve this problem, this application proposes a classification random sampling strategy for training. During the training process, for each image, in addition to the known category C, we randomly select a certain number of categories from the vocabulary to create a subset C sub For mask prediction It should be noted that for each image in the batch, different category subsets are constructed; this method enables the model to predict and learn random categories instead of fixed categories in each training iteration, thereby reducing the computational load and preventing the model from overfitting to the categories of the constructed dataset; for the predicted mask, considering that although there is a sufficiently diverse set of categories in the dataset, it is still limited; to alleviate the suppression of the model on the categories that have not been learned, this application does not use the activation function softmax for inter-class suppression; instead, this application uses the activation function sigmoid to independently calculate the loss function of different category masks. Specifically, the focal loss of the Binary Cross-Entropy (BCE) loss type is adopted and Dice loss to calculate the loss function of the predicted mask;

[0087] As an exemplary illustration, α represents the hyperparameter for balancing easy samples and hard samples. When and is close to 0, or and is close to 1, the corresponding sample is a hard sample, and the samples other than the hard samples are easy samples;

[0088] In this optional embodiment, as mentioned above, this application hopes to overcome the noise in the pseudo-mask, which will affect the segmentation performance of the model; at the same time, the goal of this application is to enable the model to learn object recognition and scene recognition capabilities independent of categories; to achieve this, this application uses images that are similar to the original scene but lack the target category as counterfactual images for self-supervised training; Figure 3 Schematic diagram of the counterfactual image pair proposed for this embodiment Figure 3 The first row of Figure 3 shows the original image, the second row shows the counterfactual image, and the third row shows the mask. Among them, Figure 3It can be seen that by using the counterfactual image to emphasize the recognition target of the original image, a mask result with less noise can be obtained; wherein, the class symbols p of the original image and the counterfactual image are extracted from the image encoder of the open world segmentation model clx ,p co To calculate the cosine similarity, we can use the loss function based on cosine similarity. To conduct comparative training;

[0089] In this optional embodiment, for the constructed dataset, a category random sampling strategy and counterfactual contrast training are proposed; these methods effectively enable the model to learn open-world segmentation capabilities;

[0090] As an exemplary illustration, the calculated loss values ​​are added and back-propagated to train and optimize the open-world detection model, and any image to be segmented and the class name to be segmented are input into the optimized open-world detection model, and the open-world detection model outputs the segmentation result; as an exemplary illustration, an image including a cat, a dog, a house and a tree and "cat" and "house" are input into the optimized open-world detection model, and the open-world detection model outputs the result of segmenting the cat and the house in an image including a cat, a dog, a house and a tree; wherein, the image encoding of the image to be segmented is obtained by the image encoder of the open-world detection model, and the text encoder of the open-world detection model is used to encode the category to be segmented, and the semantic segmentation map is obtained by calculating the similarity between the image encoding and the category encoding, and then the segmentation result is obtained.

[0091] In summary, this embodiment solves the problem that due to the lack of pixel-level labeled data, the segmentation ability of the open-world segmentation model is relatively limited compared with the closed-set semantic segmentation paradigm. By diffusion generation, a dataset with pixel-level labels is constructed at low cost, providing effective, low-cost pixel-level annotation for the open-world style model, thereby improving the segmentation ability; it solves the problem that the generated dataset mask is noisy, affecting training, has limited categories, and is prone to overfitting, and generates counterfactual image pairs for counterfactual comparative learning, which solves the overfitting problem caused by the noise caused by the mask quality problem of the generated image and the limited categories; it solves the problem that the vocabulary has too many categories and few effective training gradients, and uses a category random sampling strategy for training to maintain the stability of feature expression, restore its original feature expression, and reduce interference.

[0092] Example 2

[0093] This embodiment is based on the open domain segmentation method based on diffusion model counterfactual image generation proposed in Example 1, and proposes the following performance comparison example:

[0094] On the public datasets Pascal VOC 2012, Pascal Context, and Coco, compared with previous open-world segmentation models such as GroupViT, ViewCo, and MixReorg, the optimized open-world segmentation model (MagicSeg) in this application achieved the best results. The comparison results are shown in Table 1.

[0095] Table 1

[0096]

[0097] Our optimized open-world segmentation model (MagicSeg) is compared with several existing methods for generating semantic segmentation datasets. The comparison results are shown in Table 2. Compared with other methods, MagicSeg has the advantage of not requiring the construction of datasets for different downstream tasks. It can be seen that our method (Ours) is comparable to or even stronger than these methods that specifically design datasets for downstream tasks.

[0098] Table 2

[0099] method Model Pascal Voc COCO DiffuMask MaskFormer(R50) 57.4 - Dataset Diffusion DeepLabV3(R50) 61.6 32.4 Dataset Diffusion DeepLabV3(R101) 64.8 34.2 Dataset Diffusion MaskFormer(R50) 60.2 31.0 Ours MagicSeg 62.9 40.2

[0100] Figure 4 This is a visual comparison diagram proposed in this embodiment. Figure 4 The visualization of the method proposed in this application (MagicSeg) on ​​the Pascal VOC and COCO datasets is shown, and the comparison is made with GroupViT. MagicSeg demonstrates impressive segmentation capabilities for different categories. Compared with GroupViT, MagicSeg shows significant segmentation capabilities for images containing multiple categories and effectively captures fine details, even for small objects.

[0101] Figure 5 This is an example diagram of the visualization of the open world segmentation results proposed in this embodiment; Figure 5 The open-world segmentation results predicted by MagicSeg are visualized. It can be seen that MagicSeg can segment categories that have not been seen in training, which shows that our method has strong segmentation capabilities in open-world segmentation.

[0102] Example 3

[0103] This embodiment proposes an open domain segmentation system based on diffusion model counterfactual image generation, which is used to implement the open domain segmentation method based on diffusion model counterfactual image generation proposed in Example 1.

[0104] Figure 6This is an overall framework diagram of the open domain segmentation system based on diffusion model counterfactual image generation in this embodiment.

[0105] The open domain segmentation system based on diffusion model counterfactual image generation includes:

[0106] A vocabulary acquisition module is used to acquire a vocabulary consisting of several categories of words;

[0107] a text prompt generation module, configured to generate text prompts based on the vocabulary using a large language model;

[0108] A counterfactual text generation module for creating counterfactual text based on text prompts;

[0109] An original image generation module, used to generate an original image based on a text prompt using a diffusion model;

[0110] A counterfactual image generation module, which is used to generate counterfactual images based on counterfactual text using a diffusion model;

[0111] A mask extraction module, which extracts ground-truth masks from raw images using an open-world detection model and an interactive segmentation model;

[0112] A model acquisition module for acquiring an open-world segmentation model for predicting image masks;

[0113] A model training module is used to optimize the training of the open-world segmentation model using real masks and counterfactual images. When the open-world segmentation model's ability to predict image masks reaches an optimal level or the number of training times reaches a preset value, the training is stopped to obtain an optimized open-world segmentation model.

[0114] This embodiment proposes a computer device including a memory and a processor, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor performs the steps of the open domain segmentation method based on diffusion model counterfactual image generation described in Example 1.

[0115] It can be understood that the open domain segmentation system and computer device based on diffusion model counterfactual image generation in this embodiment improve the method of Example 1. The options in the above-mentioned Example 1 are also applicable to this embodiment, so they will not be repeated here.

[0116] The same or similar reference numerals correspond to the same or similar components;

[0117] The terms used in the drawings to describe positional relationships are for illustrative purposes only and should not be construed as limiting the present embodiment.

[0118] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. An open domain segmentation method based on counterfactual image generation using a diffusion model, characterized in that The following steps are involved: Obtain a vocabulary consisting of several categories of words; generating text prompts based on the vocabulary using a large language model; Create counterfactual text based on text prompts; Based on the text prompt, the original image is generated using the diffusion model; Based on counterfactual text, a diffusion model is used to generate counterfactual images; Leveraging open-world detection models and interactive segmentation models to extract ground-truth masks from raw images; Get an open-world segmentation model for predicting image masks; Optimizing and training the open-world segmentation model using the real mask and the counterfactual image, and stopping the training when the open-world segmentation model's ability to predict the image mask reaches an optimal level or the number of training cycles reaches a preset value, thereby obtaining an optimized open-world segmentation model; The steps of creating counterfactual text based on the text prompt include: Replacing the class name of the vocabulary in the text prompt with a replacement sentence to form a counterfactual text; Wherein, the replacement statement includes: "nothing"; The step of optimizing and training the open-world segmentation model using the true mask and the counterfactual image comprises: Extract m categories of words from the vocabulary to create a vocabulary subset , subset the vocabulary The original image is input into the open-world segmentation model, and the open-world segmentation model outputs the predicted mask; Design the first predicted mask loss function based on the real mask and the predicted mask and the second predicted mask loss function ; Design a comparative training loss function based on counterfactual images and original images ; Based on the loss function , loss function and loss function , design the total loss function , iteratively solve the total loss function , until the total loss function When the minimum value is reached or the number of iterations reaches the preset number, the iteration is stopped, the optimization of the open world segmentation model is completed, and the optimized open world segmentation model is obtained; Among them, the total loss function The expression is: 。 2. The open domain segmentation method based on diffusion model counterfactual image generation according to claim 1, characterized in that The expression of the vocabulary is: Where, represents the first-class words in the vocabulary, The first Class vocabulary, The total number of categories of words in the vocabulary; Based on the vocabulary, the steps of generating text prompts using a large language model include: Extract from vocabulary Class vocabulary, the extracted The class name and conditional statement of the class vocabulary are input into the large language model, and the large language model outputs a text prompt; among them, The size of is the preset value, and , the conditional statements include: "the scenes and details of these examples are as diverse as possible" and / or "the features of the objects are as detailed as possible." 3. The open domain segmentation method based on diffusion model counterfactual image generation according to claim 1 or 2, characterized in that: Using the open-world detection model and the interactive segmentation model, the steps to extract the ground-truth mask from the original image include: The class names of the words in the original image and the text prompt are input into the open-world detection model. The open-world detection model recognizes the objects corresponding to the class names of the words from the original image and outputs the recognition bounding box. The interactive segmentation model extracts a ground-truth mask from the original image based on the identified bounding box; The expression of the real mask is: Where, represents the true mask, and Represents the vocabulary Class vocabulary Hedi Class vocabulary The corresponding real mask, Indicates that the first dimension is , the second dimension is , the third dimension is A tensor of , where Indicates the height of the image corresponding to the real mask, Indicates the width of the image corresponding to the true mask.

4. The open domain segmentation method based on diffusion model counterfactual image generation according to claim 3, characterized in that The first prediction mask loss function The expression is: Where, represents the hyperparameter, Represents the category of image pixel k The corresponding real mask; Represents the category of pixel k The corresponding prediction mask.

5. The open domain segmentation method based on diffusion model counterfactual image generation according to claim 4, characterized in that: The second prediction mask loss function The expression is: Where, Represents the category of image pixel k The corresponding real mask; Represents the category of pixel k The corresponding prediction mask.

6. The open domain segmentation method based on diffusion model counterfactual image generation according to claim 5, characterized in that The contrastive training loss function The expression is: Where, represents the class symbol corresponding to the counterfactual image, represents the class symbol of the original image corresponding to the counterfactual image, Represents the modulus operation of an orientation quantity.

7. An open domain segmentation system based on diffusion model counterfactual image generation, used to implement the open domain segmentation method based on diffusion model counterfactual image generation according to any one of claims 1 to 6, characterized in that: include: A vocabulary acquisition module is used to acquire a vocabulary consisting of several categories of words; a text prompt generation module, configured to generate text prompts based on the vocabulary using a large language model; A counterfactual text generation module for creating counterfactual text based on text prompts; An original image generation module, used to generate an original image based on a text prompt using a diffusion model; A counterfactual image generation module, which is used to generate counterfactual images based on counterfactual text using a diffusion model; A mask extraction module, which extracts ground-truth masks from raw images using an open-world detection model and an interactive segmentation model; A model acquisition module for acquiring an open-world segmentation model for predicting image masks; A model training module is used to optimize the training of the open-world segmentation model using real masks and counterfactual images. When the open-world segmentation model's ability to predict image masks reaches an optimal level or the number of training times reaches a preset value, the training is stopped to obtain an optimized open-world segmentation model.

8. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, characterized in that: When the computer-readable instructions are executed by the processor, the processor performs the steps of the open domain segmentation method based on diffusion model counterfactual image generation according to any one of claims 1 to 6.

Images