Image open vocabulary semantic segmentation method and system
By constructing multiple prompt sentences and optimizing the boundary using variational encoders, diffusion models, and multi-layer attention maps, the problem of locating unknown categories in open vocabulary semantic segmentation is solved, achieving efficient and accurate segmentation without additional training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2024-01-30
- Publication Date
- 2026-07-21
AI Technical Summary
Existing methods struggle to effectively handle unknown categories and achieve accurate object localization in open-vocabulary semantic segmentation tasks, and typically require additional training phases or complex data synthesis processes.
By obtaining a predefined set of semantic category names for the image to be segmented, entity categories are extracted and multiple prompts are constructed. An initial segmentation region is generated using a variational encoder and a diffusion model. The boundary is then optimized by combining multi-layer cross-attention maps and self-attention maps to achieve segmentation.
Without requiring additional training or data synthesis, it generates complete and accurate localization regions for target categories, solving the problem of incomplete segmentation regions and improving the accuracy and completeness of segmentation.
Smart Images
Figure CN117994515B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of semantic segmentation and deep learning technology, and more specifically to an image open vocabulary semantic segmentation method and system. Background Technology
[0002] Open-vocabulary semantic segmentation performs fine-grained classification of each pixel in an image, enabling pixel-level understanding and segmentation. Traditional semantic segmentation tasks typically classify images based on predefined, fixed categories, such as people, cars, and trees, using predefined category labels. However, open-vocabulary semantic segmentation not only classifies predefined categories but also identifies unknown, novel, or rare categories in the image—that is, it identifies categories that have not been previously seen.
[0003] The challenge of open-vocabulary semantic segmentation lies in handling unknown categories and class balance, and the key to success is identifying the object categories appearing in the image and generating complete and accurate localization regions. Existing methods generally use pre-trained multimodal discriminative models (CLIPs) to achieve open-vocabulary segmentation. However, because these methods use discriminative models, they often result in the lack of important localization information or incomplete predicted regions. Another type of method uses generative models such as conditional diffusion models to achieve this task, but these methods require additional training phases or complex data synthesis processes.
[0004] Therefore, how to effectively solve the problem of difficulty in obtaining entity categories from open vocabulary lists, and how to achieve accurate localization of the target object's location, thereby reducing additional training stages or data synthesis processes, are problems that urgently need to be solved by those skilled in the art. Summary of the Invention
[0005] In view of the above problems, the present invention provides an image open vocabulary semantic segmentation method and system to at least solve some of the technical problems mentioned in the background art.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] On one hand, embodiments of the present invention provide an image open vocabulary semantic segmentation method, including the following steps,
[0008] S1. Obtain the image to be segmented, and the predefined semantic category name set corresponding to the image to be segmented;
[0009] S2. Extract entity categories from the image to be segmented, and match the entity categories with the predefined semantic category name set to obtain multiple prompts;
[0010] S3. Segment the image to be segmented based on the multiple prompts to obtain multiple initial segmentation regions;
[0011] S4. Optimize the boundaries of each of the initial segmented regions to obtain multiple segmentation score maps;
[0012] S5. Based on the multiple segmentation score images, the image to be segmented is segmented.
[0013] Further, step S2 specifically includes:
[0014] S21. Obtain the semantic description of the image to be segmented, and extract entity categories from the semantic description;
[0015] S22. Match the entity category with the predefined semantic category name set for relevance, and take the semantic category name corresponding to the relevance exceeding the preset value as the subsequent category to be segmented, thereby obtaining multiple categories to be segmented;
[0016] S23. Based on the multiple categories to be segmented, construct multiple prompt sentences corresponding to subsequent segmentation.
[0017] Furthermore, step S23 also includes: adding corresponding word weights for the category to be segmented.
[0018] Furthermore, step S3 specifically includes:
[0019] S31. The image to be segmented is downsampled using a variational encoder to obtain latent space feature variables;
[0020] S32. Based on the latent space feature variables, obtain the query matrix;
[0021] S33. Map the multiple prompts to obtain multiple corresponding key matrices;
[0022] S34. Calculate the multi-layer cross-attention map corresponding to each prompt sentence by combining the query matrix with multiple key matrices respectively;
[0023] S35. Perform weighted fusion processing on the cross-attention maps of the tokens corresponding to the category words in each prompt sentence, and use the fused cross-attention maps as the initial segmentation regions; thereby obtaining multiple initial segmentation regions corresponding to multiple prompt sentences.
[0024] Furthermore, step S32 specifically includes:
[0025] S321. Perform diffusion model noise addition processing on the latent space feature variables; expressed as:
[0026]
[0027] Where z represents the latent space feature variable; z t This represents the latent space feature variables after adding noise at time step t; α t The weight parameter represents time t; ε represents random noise that follows a Gaussian distribution.
[0028] S322. Extract features from the latent space feature variables after adding noise to the diffusion model to obtain latent space features;
[0029] S323. The latent space features are mapped using a linear layer to obtain the query matrix Q; represented as:
[0030]
[0031]
[0032] Where l represents a linear layer; Represent latent space features; H represents the real number field; W represents the feature map height; C represents the feature map width; and C is the number of channels in the feature map.
[0033] Furthermore, step S33 specifically includes:
[0034] S331. Encode the multiple prompts using a text encoder to obtain multiple corresponding mapping results; the mapping results are represented as follows:
[0035]
[0036] Where, τ θ (P) represents the mapping result; N represents the length of the prompt; D represents the dimension of the latent space mapping; P represents the prompt.
[0037] S332. The multiple mapping results are processed by a linear layer to obtain multiple corresponding key matrices; the key matrix is represented as follows:
[0038] K = l K (τ θ (P))
[0039] Where K represents the key matrix; l represents the linear layer.
[0040] Further, in step S34, the multi-layer cross-attention map is represented as follows:
[0041]
[0042]
[0043] Among them, A cross This represents a multi-level cross-attention graph; Q represents the query matrix; K represents the key matrix; T represents the transpose. H represents the real number field; W represents the feature map height; N represents the feature map width; d represents the length of the prompt; and Softmax(.) represents the softmax function.
[0044] Further, in step S35, the initial segmented region is represented as:
[0045]
[0046] in, This represents the c-th initial segmentation region; This represents the cross-attention graph of layer b for the token corresponding to the c-th category word; ∑ b∈B w b =1 indicates an important weight parameter.
[0047] Further, step S4 specifically includes:
[0048] S41. Obtain the corresponding multi-layer self-attention map based on the latent space features, and then perform fusion processing after interpolating the multi-layer self-attention maps to the same size; represented as:
[0049]
[0050] in, This represents the fused multi-layer self-attention map; This represents the self-attention map of layer b.
[0051] S42. Based on the fused multi-layer self-attention map, the boundary of each initial segmentation region is optimized to obtain a corresponding segmentation score map; the segmentation score map is represented as follows:
[0052]
[0053]
[0054] Among them, M c This represents the c-th segmentation fraction; the norm(·) function represents max-min normalization, ensuring that the value range of the segmentation fraction is between 0 and 1; vec(·) represents matrix-vector operations; This represents the c-th initial segmentation region.
[0055] Further, step S5 specifically includes:
[0056] The multiple segmentation fraction images are interpolated to the same size as the image to be segmented to obtain the interpolated segmentation fraction image M'. c , The interpolated segmentation map M' is then subjected to a threshold. c The image to be segmented is classified to obtain multiple corresponding segmentation masks; the image to be segmented is then segmented based on the multiple segmentation masks; the segmentation mask is represented as:
[0057]
[0058]
[0059]
[0060] Where C' represents the set of categories to be segmented; M' represents the segmentation mask corresponding to the current category k to be segmented; k M' represents the segmentation score map corresponding to the k-th category to be segmented; h Let represent the segmentation score image corresponding to the h-th category to be segmented; [i,j] represents the i-th row and j-th column of the segmentation mask matrix; σ represents the threshold. This represents the segmentation mask corresponding to the background.
[0061] On the other hand, embodiments of the present invention provide an image open vocabulary semantic segmentation system, which applies the above-described method and includes: an acquisition module, a prompt sentence construction module, a semantic segmentation module, a segmentation region enhancement module, and a segmentation module;
[0062] The acquisition module is used to acquire the image to be segmented and the predefined semantic category name set corresponding to the image to be segmented;
[0063] The prompt sentence construction module is used to extract entity categories from the image to be segmented and match the entity categories with the predefined semantic category name set to obtain multiple prompt sentences.
[0064] The semantic segmentation module is used to segment the image to be segmented based on the multiple prompts to obtain multiple initial segmentation regions.
[0065] The segmentation region enhancement module is used to optimize the boundary of each initial segmentation region to obtain multiple segmentation score maps.
[0066] The segmentation module is used to segment the image to be segmented based on the multiple segmentation score images.
[0067] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses an image open vocabulary semantic segmentation method and system, which has the following beneficial effects:
[0068] This invention can generate complete and accurate localization regions for target categories without requiring additional training phases or data synthesis processes, thus enabling complex semantic segmentation and discrimination tasks.
[0069] This invention makes full use of image-side information to further complete the positioning area and solves the problem of incomplete area segmentation in existing methods.
[0070] This invention can accurately capture the target categories appearing in an image scene.
[0071] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention.
[0072] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0073] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0074] Figure 1 This is a schematic diagram of the image open vocabulary semantic segmentation method provided in an embodiment of the present invention.
[0075] Figure 2 This is a visual illustration of different prompt sentence construction methods provided in the embodiments of the present invention.
[0076] Figure 3 This is a visualization diagram of cross-attention maps at different layers provided in an embodiment of the present invention.
[0077] Figure 4 This is a visual schematic diagram of different segmentation generation schemes provided in the embodiments of the present invention.
[0078] Figure 5 This is a schematic diagram of the image open vocabulary semantic segmentation system framework provided in an embodiment of the present invention. Detailed Implementation
[0079] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0080] See Figure 1 As shown in the figure, this invention discloses an image open vocabulary semantic segmentation method, which includes the following steps:
[0081] S1. Obtain the image to be segmented, and the predefined semantic category name set corresponding to the image to be segmented;
[0082] S2. Extract entity categories from the image to be segmented, and match the entity categories with a predefined set of semantic category names to obtain multiple prompts;
[0083] S3. Segment the image to be segmented based on multiple prompts to obtain multiple initial segmentation regions;
[0084] S4. Optimize the boundaries of each initial segmented region to obtain multiple segmentation score maps;
[0085] S5. Based on multiple segmentation score images, segment the image to be segmented.
[0086] The method provided in this embodiment of the invention belongs to a two-dimensional image segmentation method. Through this method, more accurate open vocabulary semantic segmentation results can be achieved without additional data labels and model training.
[0087] The following sections will provide a detailed explanation of each of the above steps.
[0088] In step S1 above, the image to be segmented, x∈R, is first obtained. H×W×3 And obtain the predefined semantic category name set corresponding to the image to be segmented. Where n represents the total number of semantic category names in the predefined semantic category name set C; c i This represents the name of the i-th semantic category; the input information includes the pixel RGB colors of the image to be segmented.
[0089] In step S2 above, entity categories are extracted from the image x to be segmented, and these entity categories are matched with a predefined semantic category name set C to obtain multiple prompts; specifically, the following content is included:
[0090] S21. Obtain the semantic description of the image to be segmented x, and extract the entity category from the sentence of the semantic description;
[0091] S22. Match the entity category with the predefined semantic category name set C for relevance, and take the semantic category name corresponding to the relevance exceeding the preset value as the subsequent category to be segmented; thereby obtaining multiple categories to be segmented, that is, obtaining the set C' of categories to be segmented;
[0092] S23. Based on multiple categories to be segmented, construct multiple prompt sentences P corresponding to subsequent segmentation. Specifically, this includes: First, constructing an initial prompt sentence for each category to be segmented. The template sentence for this initial prompt sentence is "a photo of {}", where {} represents the name of the category to be segmented; thus, multiple prompt sentences can be obtained; see [link to documentation]. Figure 2 As shown, an additional word weight, represented as "{category}++", is added to the name of the category to be segmented to enhance the subsequent segmentation results for the target class. Then, in this step, since a complete and detailed prompt sentence helps improve the subsequent segmentation results, the template sentence can be further completed based on the template sentence filled with the category to be segmented and the image to be segmented, to obtain the final more detailed prompt sentence P.
[0093] In step S3 above, the image to be segmented is segmented based on multiple prompts P to obtain multiple initial segmentation regions; specifically, this includes the following:
[0094] S31. The latent space feature variable z = ∈(x) is obtained by downsampling the image to be segmented using a variational encoder (VAE Encoder).
[0095] S32. Obtain the query matrix based on latent space feature variables; specifically:
[0096] S321. Apply a diffusion model to the latent space feature variables to add noise; expressed as:
[0097]
[0098] Where z represents the latent space feature variable after initial downsampling by the variational encoder; z t The noisy step t represents the latent space feature variable after adding noise for a time step of t. In this embodiment of the invention, the noise step t can be set to 100. α t The weight parameter represents time t; ε represents random noise that follows a Gaussian distribution.
[0099] S322. Extract features from the latent space feature variables after adding noise to the diffusion model to obtain latent space features;
[0100] S323. The latent space features are mapped through a linear layer to obtain the query matrix Q (i.e., the query matrix); represented as:
[0101]
[0102]
[0103] Where l represents a linear layer; Represent latent space features; H represents the real number field; W represents the feature map height; C represents the feature map width; and C is the number of channels in the feature map.
[0104] S33. Map multiple prompts to obtain multiple corresponding key matrices;
[0105] S331. Encode multiple prompts using a text encoder to obtain multiple corresponding mapping results; the mapping results are represented as follows:
[0106]
[0107] Where, τ θ (P) represents the mapping result; N represents the length of the prompt; D represents the dimension of the latent space mapping; P represents the prompt.
[0108] S332. The above mapping results are processed by a linear layer to obtain multiple corresponding key matrices; the key matrix is represented as follows:
[0109] K = l K (t θ (P))
[0110] Where K represents the key matrix; l represents the linear layer.
[0111] S34. Calculate the multi-layer cross-attention map corresponding to each prompt sentence by combining the query matrix with multiple key matrices respectively:
[0112]
[0113]
[0114] Among them, A cross This represents a multi-level cross-attention graph; Q represents the query matrix; K represents the key matrix; T represents the transpose. The domain is defined as real numbers, indicating that the range of the preceding variables is real numbers; H represents the feature map height; W represents the feature map width; d represents the feature channel length; Softmax(.) represents the softmax function; N represents the length of the prompt, i.e., the number of words in the prompt. The length of the resulting cross-attention map is the same as the length of the prompt, i.e., one word token corresponds to one H*W feature map. For example, suppose a prompt is "a photo of car", which has 4 words, then the length of the prompt is 4. Since each word in each prompt will generate a cross-attention map, the prompt will generate 4 layers of cross-attention maps with different resolutions. In subsequent steps, only the cross-attention map corresponding to the category is extracted, i.e., the cross-attention map corresponding to the token after mapping the word "car". Thus, if there are n sentences, i.e. n categories to be segmented, the cross-attention map corresponding to the category of each sentence must be found separately.
[0115] S35. From the multi-layer cross-attention map in each prompt sentence, extract the cross-attention map of the token corresponding to the category word in that prompt sentence, and perform weighted fusion processing on the cross-attention map of the token corresponding to the category word in each prompt sentence with different weights, and use the fused cross-attention map as the initial segmentation region; thus obtaining multiple initial segmentation regions corresponding to multiple prompt sentences; specifically:
[0116] Embodiments of the present invention Figure 3 There are four layers of cross-attention maps in the middle; through Figure 3 It is known that cross-attention maps of different layers can capture different levels of semantic information. Deeper and smaller cross-attention maps can better capture the localization information of the target object, while shallower and larger ones can better capture the detailed information of the object. Therefore, in this embodiment of the invention, the multi-layer cross-attention maps of the corresponding category token c are weighted and fused with different weights w1.
[0117]
[0118] in, This represents the c-th initial segmentation region; This represents the cross-attention graph of layer b for the token corresponding to the c-th category word; ∑ b∈B w b =1 indicates a significant weight parameter; b indicates the b-th layer in UNet corresponding to the cross-attention map calculation position, with a total of B layers; in this method, the weights are [0.3, 0.5, 0.1, 0.1] according to the attention map resolution from smallest to largest, and the weighted result is better than the result of averaging 4 layers; see [link to relevant documentation]. Figure 3As shown, in this invention, the cross-attention maps of four layers with different sizes (8×8, 16×16, 32×32, 64×64) in the UNet network are all interpolated to the same size (64×64) before being weighted and fused; the fused cross-attention map This is the initial segmented region.
[0119] In step S4 above, the boundaries of each initial segmented region are optimized to obtain multiple segmentation score maps; specifically, this includes the following:
[0120] S41. Obtain the corresponding multi-layer self-attention maps based on the latent space features, interpolate the multi-layer self-attention maps to the same size, and then perform fusion processing. The fusion process is similar to the cross-attention map fusion process; represented as:
[0121]
[0122] in, This represents the fused multi-layer self-attention map; Let K represent the self-attention map of layer b. The method for obtaining this self-attention map is the same as that for obtaining the cross-attention map described above. The difference is that the key matrix K of the self-attention map is obtained from the latent space features. This is unrelated to the prompt; 'b' indicates that the self-attention map calculation location corresponds to the b-th layer in UNet, and there are a total of B layers. This method only uses the self-attention map corresponding to the 64×64 latent space features.
[0123] S42. Based on the fused multi-layer self-attention map, the boundaries of each initial segmentation region are optimized to obtain the corresponding segmentation score map; see [link / reference]. Figure 4 As shown, in this embodiment of the invention, only the self-attention map corresponding to the latent space features of size 64×64 was selected because higher resolution can capture more image details. If the results of four self-attention layers are used simultaneously, the low-resolution result will actually lead to blurred object boundaries. This result is then used to further optimize the initial segmentation region to obtain the final segmentation score map. This segmentation score map is represented as follows:
[0124]
[0125]
[0126] Among them, M c This represents the c-th segmentation fraction; the norm(·) function represents max-min normalization, ensuring that the value range of the segmentation fraction is between 0 and 1; vec(·) represents matrix-vector operations; This represents the c-th initial segmentation region.
[0127] In step S5 above, based on the multiple segmentation score images obtained above, the image to be segmented is segmented, and the segmentation regions for all possible categories of the image to be segmented are output; specifically, this includes the following:
[0128] Multiple segmentation fraction images are interpolated to match the size of the image to be segmented, i.e., interpolated from 64*64 to h*w size, to obtain the interpolated segmentation fraction image M'. c , The interpolated segmentation map M' is then subjected to a threshold. c The image is classified to obtain multiple corresponding segmentation masks; these segmentation masks are then used to segment the image to be segmented; the segmentation mask is represented as:
[0129]
[0130]
[0131]
[0132] Where C' represents the set of categories to be segmented; M' represents the segmentation mask corresponding to the current category k to be segmented; k M' represents the segmentation score map corresponding to the k-th category to be segmented; h Let M' represent the segmentation score map corresponding to the h-th category to be segmented; the segmentation score map value of the current category k to be segmented is greater than the segmentation score map values of all other categories h to be segmented, i.e., M' k [i,j]>M' h [i,j]; [i,j] represents the i-th row and j-th column of the segmentation mask matrix; σ represents the threshold; This represents the segmentation mask corresponding to the background;
[0133] For the current category k to be segmented, if the value of the segmentation score map corresponding to the current category k to be segmented is greater than the value of the segmentation score map of other categories in the i-th row and j-th column, and this value is greater than the threshold, then the final segmentation mask of the current category k to be segmented is 1 at the position of that pixel; otherwise, it is 0.
[0134] If a pixel is not activated in any of the categories to be segmented, meaning its activation value is less than the threshold in all score maps of all categories, then the pixel belongs to the background class. Assuming there are c categories to be segmented, we will ultimately obtain c+1 masks: c category masks and 1 background mask. In each mask, a region with a value of 1 indicates that the pixel belongs to that category, and...
[0135]
[0136] In other words, a pixel can only be activated as 1 in the segmentation mask of one category. If it does not belong to any category C', it is the background category.
[0137] In this embodiment of the invention, after repeated experiments, it was found that setting the threshold between [0.4, 0.5] yielded better results, as shown in Table 1:
[0138] Table 1 shows the performance of this method on different datasets at different thresholds.
[0139] Object 34.27 36.23 37.4 37.9 37.4 35.8 Context 27.4 27.5 27.1 26.1 24.7 22.5 VOC 59.3 60.6 60.7 59.3 56.1 51.5
[0140] See Figure 5 As shown, this embodiment of the invention also provides an image open vocabulary semantic segmentation system, applying the above-described image open vocabulary semantic segmentation method; the image open vocabulary semantic segmentation system includes: an acquisition module, a prompt sentence construction module, a semantic segmentation module, a segmentation region enhancement module, and a segmentation module; wherein:
[0141] 1. Acquisition Module: For example, in this embodiment of the invention, the datasets used include PASCAL VOC 2012 (VOC), which contains 21 categories (1 background category), with a total of 10,582 training images and 1,449 test images; Pascal Context (Context), which contains 60 categories (including the background category); and COCO Object (Object), which contains 80 categories; VOC, Context, and Object used 1,449, 5,105, and 5,000 images respectively for testing.
[0142] The acquisition module is used to acquire the image to be segmented, as well as the predefined semantic category name set corresponding to the image to be segmented; the image to be segmented contains RGB three-channel information.
[0143] 2. Prompt Sentence Construction Module: This module uses a pre-trained BLIP model capable of constructing image scene descriptions to obtain semantic descriptions of the image to be segmented; it inputs the image... After obtaining the image description, entity categories are extracted from the description sentence; these entity categories are then matched with predefined semantic category names using the cross-modal pre-trained model CLIP for relevance matching; predefined semantic categories with high relevance are extracted as subsequent categories to be segmented; thus obtaining multiple categories to be segmented; based on these multiple categories to be segmented, multiple prompt sentences are constructed for input to the pre-trained production model, specifically including: first, based on each category to be segmented, a corresponding initial prompt sentence is constructed, the template sentence corresponding to this initial prompt sentence is "a photo of {}", where {} represents the target category name; such as Figure 2As shown, this method adds an extra word weight to the target category name, represented as "{category}++", to enhance the subsequent segmentation results for the target class. Then, since a complete and detailed prompt also helps improve the subsequent segmentation results, the template sentence filled with the category to be segmented and the image to be segmented are input back into the BLIP model to further complete the template sentence and obtain the final more detailed prompt.
[0144] 3. Semantic Segmentation Module: This semantic segmentation module receives the image x to be segmented and multiple prompts P constructed by the prompt construction module, and performs preliminary segmentation on the image x to be segmented based on the multiple prompts to obtain multiple corresponding initial segmentation regions;
[0145] Specifically: This semantic segmentation module uses Stable Diffusion v1.5 as the pre-trained conditional diffusion model framework, and its pre-trained parameters are directly loaded without additional training; the Stable Diffusion model consists of three parts: a variational encoder, a decoder, and a UNet noise prediction network; the input image to be segmented is first interpolated to a uniform size of 512×512, and then the latent space feature variable z=∈(x) of the image is obtained by downsampling through the variational encoder ∈; then the latent space feature variable is subjected to the diffusion model to add noise, and the latent space feature variable after adding noise for time step t is obtained; the relevant formulas can be found in the above description of the method.
[0146] 4. Segmentation Region Enhancement Module: This module optimizes the boundaries of each initial segmentation region, obtaining multiple segmentation score maps. It utilizes the self-correlation information of image features to further enhance and complete the initial segmentation regions. While the semantic segmentation module can locate the target object, the object boundaries are often blurry, making complete object segmentation difficult. Furthermore, the UNet network also possesses a self-attention matrix with latent space features. The query and key matrices of the attention map are calculated based on latent space features. The mapping is obtained; the relevant calculation formulas can be found in the methods described above.
[0147] 5. Segmentation Module: Based on the multiple segmentation score maps mentioned above, this module segments the image to be segmented and outputs the segmentation regions for all possible categories in the image.
[0148] Most open vocabulary semantic segmentation tasks are built on discriminative models such as CLIP. However, due to limitations in training methods, CLIP and similar models often struggle to capture complete object information in segmentation tasks. Generative models, such as conditional diffusion models, require the generation of complete objects, resulting in more comprehensive object perception and capture. Therefore, this invention can generate more complete segmentation regions compared to previous methods. Furthermore, this invention does not require any additional training data or training phases. Compared to other methods that use conditional diffusion models to achieve this task, which require additional training phases and segmentation modules, or require a significant amount of time to generate images of the categories to be segmented and extract category features, this invention only requires a single-step diffusion model denoising process to generate the final segmentation mask. Specifically, this invention solves the following three problems: (1) The difficulty in obtaining entity categories from open vocabulary. (2) By utilizing the pre-trained text-image pair information of the conditional diffusion model itself, a connection between text and pixels is established, achieving accurate localization of the target object's location and solving the problem of regional localization in multi-object or complex scenes. (3) By fully utilizing image-side information, further completion of the localization region is achieved, solving the problem of incomplete segmentation regions in existing methods.
[0149] As shown in Tables 2 and 3, the embodiments of the present invention outperform existing methods on various datasets for open-vocabulary semantic segmentation tasks. They also outperform existing methods in weakly supervised semantic segmentation tasks (where the category name of the object to be segmented is provided during segmentation, eliminating the need to manually obtain it from predefined semantic categories).
[0150] Table 2 shows the performance of different methods on different datasets for open vocabulary semantic segmentation tasks.
[0151]
[0152] Table 3. Performance of different methods on the VOC training dataset in the weakly supervised segmentation task.
[0153] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0154] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An image open-vocabulary semantic segmentation method, characterized in that, Includes the following steps: S1. Obtain the image to be segmented, and the predefined semantic category name set corresponding to the image to be segmented; S2. Extract entity categories from the image to be segmented, and match the entity categories with the predefined semantic category name set to obtain multiple prompts; S3. Segment the image to be segmented based on the multiple prompts to obtain multiple initial segmentation regions; S4. Optimize the boundaries of each of the initial segmented regions to obtain multiple segmentation score maps; S5. Based on the multiple segmentation score images, the image to be segmented is segmented; Step S5 specifically includes: The multiple segmentation score images are interpolated to the same size as the image to be segmented to obtain the interpolated segmentation score images. , Then, a threshold is applied to the interpolated segmentation image. The image to be segmented is classified to obtain multiple corresponding segmentation masks; the image to be segmented is then segmented based on the multiple segmentation masks; the segmentation mask is represented as: in, Represents the set of categories to be segmented; This represents the segmentation mask corresponding to the current category k to be segmented; This represents the segmentation score map corresponding to the k-th category to be segmented; This represents the segmentation score map corresponding to the h-th category to be segmented; This represents the i-th row and j-th column of the segmentation mask matrix; Indicates the threshold; This represents the segmentation mask corresponding to the background.
2. The image open vocabulary semantic segmentation method according to claim 1, characterized in that, Step S2 specifically includes: S21. Obtain the semantic description of the image to be segmented, and extract entity categories from the semantic description; S22. Match the entity category with the predefined semantic category name set for relevance, and take the semantic category name corresponding to the relevance exceeding the preset value as the subsequent category to be segmented, thereby obtaining multiple categories to be segmented; S23. Based on the multiple categories to be segmented, construct multiple prompt sentences corresponding to subsequent segmentation.
3. The image open vocabulary semantic segmentation method according to claim 1, characterized in that, Step S3 specifically includes: S31. The image to be segmented is downsampled using a variational encoder to obtain latent space feature variables; S32. Based on the latent space feature variables, obtain the query matrix; S33. Map the multiple prompts to obtain multiple corresponding key matrices; S34. Calculate the multi-layer cross-attention map corresponding to each prompt sentence by combining the query matrix with multiple key matrices respectively; S35. Perform weighted fusion processing on the cross-attention maps of the tokens corresponding to the category words in each prompt sentence, and use the fused cross-attention maps as the initial segmentation regions; thereby obtaining multiple initial segmentation regions corresponding to multiple prompt sentences.
4. The image open vocabulary semantic segmentation method according to claim 3, characterized in that, Step S32 specifically includes: S321. Perform diffusion model noise addition processing on the latent space feature variables; expressed as: Where z represents the latent space feature variable; This represents the latent space feature variables after adding noise at time step t; , The weight parameter represents time t; This represents random noise that follows a Gaussian distribution. S322. Extract features from the latent space feature variables after adding noise to the diffusion model to obtain latent space features; S323. The latent space features are mapped using a linear layer to obtain the query matrix Q; represented as: Where l represents a linear layer; Represent latent space features; H represents the real number field; W represents the feature map height; C represents the feature map width; and C is the number of channels in the feature map.
5. The image open vocabulary semantic segmentation method according to claim 3, characterized in that, Step S33 specifically includes: S331. Encode the multiple prompts using a text encoder to obtain multiple corresponding mapping results; the mapping results are represented as follows: in, The mapping result is represented by N; the length of the prompt is represented by D; the dimension of the latent space mapping is represented by P; S332. The multiple mapping results are processed by a linear layer to obtain multiple corresponding key matrices; the key matrix is represented as follows: Where K represents the key matrix; l represents the linear layer.
6. The image open vocabulary semantic segmentation method according to claim 3, characterized in that, In step S34, the multi-layer cross-attention map is represented as follows: Among them, A cross This represents a multi-level cross-attention graph; Q represents the query matrix; K represents the key matrix; T represents the transpose. H represents the real number field; W represents the feature map height; N represents the feature map width; d represents the length of the prompt; and Softmax(.) represents the softmax function.
7. The image open vocabulary semantic segmentation method according to claim 3, characterized in that, In step S35, the initial segmented region is represented as: in, This represents the c-th initial segmentation region; This represents the cross-attention graph at layer b corresponding to the token of the c-th category word; This indicates important weight parameters.
8. The image open vocabulary semantic segmentation method according to claim 5, characterized in that, Step S4 specifically includes: S41. Obtain the corresponding multi-layer self-attention map based on the latent space features, and then perform fusion processing after interpolating the multi-layer self-attention maps to the same size; represented as: in, This represents the fused multi-layer self-attention map; This represents the self-attention map of layer b. S42. Based on the fused multi-layer self-attention map, the boundary of each initial segmentation region is optimized to obtain a corresponding segmentation score map; the segmentation score map is represented as follows: in, This represents the c-th segmented fractional graph; The function represents the maximum-minimum normalization, ensuring that the value range of the segmentation fractional plot is between 0 and 1; This indicates operations on matrices and vectors; This represents the c-th initial segmentation region.
9. An image open-vocabulary semantic segmentation system, characterized in that, The system using the method according to any one of claims 1-8 comprises: an acquisition module, a prompt sentence construction module, a semantic segmentation module, a segmentation region enhancement module, and a segmentation module; The acquisition module is used to acquire the image to be segmented and the predefined semantic category name set corresponding to the image to be segmented; The prompt sentence construction module is used to extract entity categories from the image to be segmented and match the entity categories with the predefined semantic category name set to obtain multiple prompt sentences. The semantic segmentation module is used to segment the image to be segmented based on the multiple prompts to obtain multiple initial segmentation regions. The segmentation region enhancement module is used to optimize the boundary of each initial segmentation region to obtain multiple segmentation score maps. The segmentation module is used to segment the image to be segmented based on the multiple segmentation score images.