Segmentation method and device adapted to pixel-level large-scale unsupervised semantic segmentation task, and storage medium

Abstract

The invention discloses a segmentation method and device adapted to a pixel-level large-scale unsupervised semantic segmentation task and a storage medium, and the method comprises the steps: taking a to-be-segmented image as the input of CLIPS, and carrying out the text-image matching to judge the category semantics of the image; under the guidance of category semantics, detecting the region where each category is located by using a detection model to obtain a geometric box prompt; and taking the geometric box prompt as the input of the SAM, and allocating semantic tags to the to-be-segmented image pixel by pixel to obtain a semantic segmentation result. According to the method, the strong zero sample generalization ability of the basic model is utilized, and the knowledge of the current domain is embedded into the frame prompt of the SAM of the basic model, so that the prompt automation of the SAM is realized, and the performance of a large-scale unsupervised semantic segmentation task is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision image processing, and in particular to a segmentation method, device and storage medium adapted to large-scale pixel-level unsupervised semantic segmentation tasks. Background Art

[0002] Semantic segmentation is a fundamental task in computer vision that heavily relies on large-scale, high-quality annotated datasets. However, obtaining such datasets is labor-intensive and expensive, especially for refined or domain-specific categories such as medical image analysis. Unsupervised semantic segmentation (USS), which refers to semantic segmentation without explicitly labeled training data, remains an active and promising research area. However, its current focus is limited to specific data diversity and categories, which restricts its application in more complex scenarios. To address this issue, the large-scale unsupervised semantic segmentation (LUSS) problem was proposed to automatically assign semantic labels to pixels in millions of images containing nearly a thousand categories without human annotation supervision. The LUSS task has the following characteristics: 1) Representing both large categories and fine-grained shapes: Recognizing concepts of hundreds of categories provides valuable insights for handling challenging scenarios with high intra-class similarity and low inter-class differences, such as disguised object detection (COD), style transfer, and domains that require expert discriminative knowledge. In addition, the learned representation of fine-grained shapes is beneficial to downstream pixel-level tasks such as instance segmentation and object segmentation. 2) Scalability and generalization: By leveraging large-scale and unorganized datasets, LUSS can obtain more robust and generalizable representations, enabling it to adapt to new data distributions and complex environments. In addition, unsupervised learning methods also provide ideas for the development of unsupervised, semi-supervised, and fully supervised models by generating pseudo-labels and constructing self-supervisory signals. However, compared with the widely studied weakly supervised semantic segmentation tasks (WSS), unsupervised domain adaptive semantic segmentation (UDASS), and semi-supervised semantic segmentation (USS), LUSS faces greater challenges that limit the applicability of many existing methods. 1) Increased data diversity and complexity: LUSS covers unbalanced images from different domains, scenes, and visual features, requiring models with enhanced generalization capabilities. 2) Fine-grained distinction: The visual differences between thousands of categories may become increasingly subtle, requiring models to effectively capture and utilize detailed information to achieve accurate pixel-level segmentation. 3) Inherent uncertainty in unsupervised learning: In the absence of manually annotated ground truth, LUSS should effectively discern and represent category-specific and shape-related semantic structures in a wide range of visual data. 4) Training efficiency: Although large-scale training data enables the model to learn rich representations in an unsupervised manner, it also inevitably introduces training costs that need to be optimized to improve efficiency.

[0003] To address the problem of scarce labeled data, self-supervised learning, domain adaptation, co-training, and pseudo-labeling methods have been proposed to achieve accurate segmentation. However, these methods are not suitable for LUSS tasks due to limited categories and low data diversity. The most advanced methods for LUSS tasks learn rich shape and category representations through enhanced representation learning and pixel-level label generation strategies based on pixel attention. However, their performance drops significantly as the number of categories increases from 50 to 919. In addition, their pixel-level feature description and segmentation capabilities drop sharply as the size of objects decreases. In general, due to the huge challenges faced by LUSS, the current state-of-the-art benchmark performance is relatively low, and there is still great potential for improvement.

[0004] We present a comprehensive analysis of the internal structures of current state-of-the-art methods for the LUSS task. The expressive power of PASS is closely tied to two main structures: the feature extraction backbone and the self-supervised representation learning module. Improving the former requires a more robust pre-trained backbone model, which means that LUSS requires more extensive prior knowledge to better understand large-scale category concepts. The latter aims to design self-learning strategies to enhance the pixel-level representation capabilities in image-level methods and the instance-level category representation capabilities in pixel-level methods, which are significantly affected by implicitly constructed supervisory signals.

[0005] In recent years, extensive pre-training on large-scale datasets has enabled visual grounding models (VFMs) such as CLIP and SAM to exhibit strong general knowledge, enabling them to successfully transfer across various downstream tasks. This raises the question: why not leverage the knowledge in pre-trained VFMs in the LUSS task? To the best of our knowledge, this direction has not received enough attention in the context of LUSS and deserves further exploration. VFMs such as SAM perform suggestable segmentation, but it differs from semantic segmentation in two aspects: the generated masks are unlabeled, and SAM relies on both spatial cues (points, boxes) and semantic cues (text). Although recent studies have proposed VFM variants such as SAMCLIP, ZegClip, and Semantic-SAM for semantic segmentation tasks, the process of fine-tuning on additional data for retraining for LUSS tasks presents significant challenges in terms of human supervision, manual hint design, and computational resources. Therefore, these methods are not acceptable for LUSS. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a segmentation method, device and storage medium that are suitable for large-scale unsupervised semantic segmentation tasks at the pixel level in view of the shortcomings of the existing technology, so as to improve the performance of large-scale unsupervised semantic segmentation.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: a segmentation method adapted to large-scale pixel-level unsupervised semantic segmentation tasks, comprising the following steps:

[0008] S1. Use the image to be segmented as the input of CLIPS and perform text-image matching to determine the category semantics of the image.

[0009] S2. Under the guidance of the category semantics, the detection model is used to detect the area where each category is located to obtain a geometric box prompt;

[0010] S3. Use the geometric box prompt as the input of SAM, assign semantic labels to the image to be segmented pixel by pixel, and obtain semantic segmentation results.

[0011] At present, the main technologies for pixel-level large-scale semantic segmentation tasks are self-supervised learning, domain adaptation, collaborative training, and pseudo-labeling methods. As the number of segmentation categories increases by thousands, the shape and category feature representation capabilities of these methods drop sharply and cannot meet performance requirements. The present invention utilizes the powerful zero-sample generalization ability of the basic model and embeds the knowledge of the current domain into the box prompt of the basic model SAM, realizing the prompt automation of SAM and greatly improving the performance of large-scale unsupervised semantic segmentation tasks.

[0012] The CLIPS comprises an image encoder and a text encoder connected in sequence; the image encoder takes a set of image patch embeddings as the Lth i+1 Input to the Transformer layer, with a learnable [CLS] token: y=Head * ([x N ]); where Ei is the feed L i+1 [CLS] embedding of layers, [.,.] indicates stack and concatenation operations, * indicates frozen parameters, Indicates that the parameters of the Li layer network are frozen, and Head embeds the [CLS] of the last layer into x N The classification head projects into the probability distribution of the predicted class, where N is the number of Transformer layers.

[0013] In step S2, the specific process of generating the geometric frame prompt includes:

[0014] 1) Sort all the positive geometric points generated by CLIPS and select the middle point as the initial point;

[0015] 2) Find all points within the radius of the initial point. If the number of points away from the initial point is less than Minpts, the initial point is marked as noise; otherwise, the initial point is marked as a core sample and a new cluster label is assigned;

[0016] 3) Visit all neighbors of the initial point and assign cluster labels to the points that are not assigned to clusters; if they are core samples, visit their neighbors in turn, and so on; the cluster gradually increases until there are no core samples within the radius of the cluster;

[0017] 4) Select another point that has not been visited and repeat steps 1) to 3) until all points are clustered into M clusters;

[0018] 5) Traverse each cluster and find the upper left and lower right coordinates in each cluster as the pseudo-truth box generated by this cluster point, that is, obtain the geometric box prompt.

[0019] In step S3, SAM includes a point generation branch and a box generation branch. The loss function expression of the box generation branch is:

[0020]

[0021] Among them, C i is the target class, is the matching loss of the box hint regulator, N is the number of training samples, y,

[0022] ∈[0,1] 4 It is a four-dimensional vector composed of the center coordinates of the detection box and its height and width relative to the image size. y represents the pseudo-truth box generated by the point generation branch. represents the geometric hint box in the pseudo-truth box generation branch, L iou is the IOU loss, ||.||1 is the L1 regularization loss, λ L1 is the weight coefficient;

[0023] Loss of masks generated by point generation branch and box generation branch for:

[0024]

[0025] z, are the pixel values ​​of the predicted masks in the point generation branch and the box generation branch, respectively, and α and β are the balance coefficients of the focal loss and the dice loss in the mask loss.

[0026] The semantic segmentation result Output is expressed as: Output = SAM (I, Prompt text ,Prompt box ); Prompt text and Prompt box are automatically generated text prompts and geometric box prompts, and I is the image to be segmented.

[0027] As an inventive concept, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the steps of the above method.

[0028] As an inventive concept, the present invention also provides a computer-readable storage medium having a computer program / instruction stored thereon; characterized in that the computer program / instruction implements the steps of the above method when executed by a processor.

[0029] As an inventive concept, the present invention also provides a computer program product, including a computer program / instruction; when the computer program / instruction is executed by a processor, the steps of the above method are implemented.

[0030] Compared with the prior art, the present invention has the following beneficial effects: the present invention utilizes the powerful zero-sample generalization capability of the basic model and embeds the knowledge of the current domain into the box prompt of the basic model SAM, thereby realizing the prompt automation of SAM and greatly improving the performance of large-scale unsupervised semantic segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flowchart of a large-scale unsupervised semantic segmentation method based on an automatic prompting large model provided by an embodiment of the present invention;

[0032] Figure 2 A flowchart of specific sub-steps of step S10 in the large-scale unsupervised semantic segmentation method based on automatic prompting of a large model provided in an embodiment of the present invention;

[0033] Figure 3 A flowchart of specific sub-steps of step S20 in the large-scale unsupervised semantic segmentation method based on automatic prompting of a large model provided in an embodiment of the present invention;

[0034] Figure 4 The structure of the semantic text adjuster provided by the embodiment of the present invention is that in the training of the downstream semantic segmentation task, only the parameters of the semantic text adjuster and the linear head are trainable, while the entire CLIPS remains frozen. The semantic text adjuster inserts a set of learnable vectors in the encoder layer of each image transformer to achieve cross-domain adaptation of CLIPS to distinguish a large number of category concepts.

[0035] Figure 5 The point prompt provided in the embodiment of the present invention is used as a supervisory signal for the box prompt. The point prompt contains the knowledge in CLIPS and can supplement the missing information of GroundingDINO. After removing the noise points, the remaining valid points are converted into pseudo-true values ​​of the box prompt generated by GroundingDINO.

[0036] Figure 6 The detailed structure of the SAM lightweight mask decoder provided by the embodiment of the present invention is as follows: The image embedding and the hint embedding are updated through cross attention in the mask decoder.

[0037] Figure 7 The structure of the box prompt regulator provided by the embodiment of the present invention. The box prompt regulator is used to adjust the box prompts generated by Grounding DINO, and correct the behavior of Grounding DINO by learning the representation correlation of SAM and CLIPS, so as to generate box prompts that obey the target text prompts. At this stage, CLIPS, Grounding DINO and SAM are all in a frozen state, and only the box prompt regulator is trained;

[0038] Figure 8 This is an overall structural diagram of the task segmentation model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The embodiment of the present invention chooses to freeze all Visual Foundation Model (VFM) parameters and use its rich knowledge to identify large category concepts in the large-scale unsupervised semantic segmentation (LUSS) task. In the embodiment of the present invention, we try to use the frozen segmentation everything model (SAM) that does not require training as a segmentation tool, but how to automatically generate SAM prompts is a new problem. Other popular VFMs can be integrated into a prompt generation architecture that does not require training. However, in the LUSS task, as the number of categories increases to thousands, some methods have difficulty handling the similarity relationship between classes and may produce noisy prompts. These construction methods can only achieve satisfactory results when predefined labels are provided. In addition, we found that the performance of the method based on the automatic prompt large model is highly dependent on the quality of the prompts used, rather than the quantity. Adding more types of prompts may not necessarily improve the segmentation performance because the noise level will increase. Therefore, it is necessary to explore the potential of VFM and provide a high-quality cue generation architecture for semantic segmentation tasks. It is also necessary to use a large amount of data to perform targeted unsupervised optimization on the domain adaptation problem of VFM in LUSS tasks, which is crucial to further improve the performance of LUSS tasks.

[0041] Example 1

[0042] See also Figures 1 to 8 The embodiment of the present invention provides a large-scale unsupervised semantic segmentation method based on automatic prompting large model, which specifically includes the following steps:

[0043] Step S10, image-level semantic hint generation. Perform text-image matching on the adjusted CLIPS input to the image to be segmented to determine the category semantics T of the image;

[0044] Wherein, step S10 includes the following sub-steps:

[0045] Step S11, image-text feature extraction is performed on the segmented image. There are two encoders in CLIPS: the image encoder is designed to map high-dimensional images to low-dimensional embedding space, which can be a CNN architecture such as ResNet50 or a visual transformer such as ViT; the text encoder adopts transformer and is designed to generate text representations from natural language. For classification tasks, the [class] token is first expanded into a template, such as "a photo of [class]", and then the sentence is converted into a unique digital ID through lowercase byte-pair encoding technology. To facilitate small batch training, each text sequence has a fixed length of 77 and carries [SOS] and [EOS] tokens. After that, the ID is mapped to a 512-D word embedding vector, which is then passed to the transformer to derive the weight vector. At the same time, the image encoder extracts image features and suppresses redundant noise. After calculating the similarity between the image features and the text weight vector, the text with a higher score greater than the set threshold (empirically set to 0.5) is selected as the target category semantics T0 of the image. Essentially, in downstream tasks, the downstream classification performance depends on the discriminative power of features extracted by image and text encoders. The functionality of CLIPS can be expressed as:

[0046] T0=CLIPS(I,Classes)

[0047] The input image to be segmented is I, the category text to be segmented is Classes, and the target category semantics of the image is T0.

[0048] Step S12, text cue regulator construction. For the image target semantics generated in S11, due to the insufficient cross-domain adaptation of CLIPS for downstream tasks, the determined target semantics T0 is inaccurate. Therefore, in order to solve the above problem, we introduce a text cue adjustment module. When we separately adjust the text encoder to adapt to the LUSS task through text cue learning technology, we find that suboptimal results occur because the gap between the pre-trained CLIPS and the target domain semantic segmentation task is not narrowed small enough. Therefore, we introduce our semantic text regulator module from the perspective of visual cue learning. For an image encoder with N layers, the input image I of height h and width w is first divided into m patches of fixed size and then embedded into a d-dimensional latent space as shown below:

[0049]

[0050] The entire image encoder can be represented by taking the set of image patch embeddings as L i+1 Input to the Transformer layer, with a learnable [CLS] token:

[0051]

[0052] y=Head * ([x N ]);

[0053] In the formula To feed L i+1 [CLS] embedding of layers, is the image embedding set, [.,.] represents stacking and concatenation operations. * represents frozen parameters. represents L i The layer network parameters are frozen. Head is to embed the [CLS] of the last layer into x N The classification head projects into the probability distribution of the predicted class. * Indicates that the classification head parameters are frozen, and y is the feature vector output by the Head layer. Figure 6 As shown, to assign a value to each image encoder layer, our prompt text regulator inserts a learnable prompt At the same time, the original parameters are frozen, and its mechanism can be expressed as:

[0054]

[0055] y=Head ○ ([x N ])

[0056] Among them, O means that the function is trainable, Indicates that the learnable prompt inserted in layer i-1 can be trained, Head ○ Indicates that the parameters of the Head classification can be trained. It should be noted that Z i It is only used as a learnable parameter to adjust the target text prompt generated by CLIPS, which is discarded and does not flow into the next layer. We wrap up the above process and represent it as TextTuner(.).

[0057] Step S13, the box supervision signal and the pixel-level supervision signal train the text prompt regulator. Both CLIPS and GroundingDINO branches can generate prompts for SAM, which are points and boxes respectively. However, the prompts input to SAM have contradictory properties of knowledge and noise. Due to the generalization advantage and cross-domain adaptation bottleneck, the prompts introduce noise while introducing knowledge. This embodiment found that in terms of noise, the point prompts of SAM introduce more noise than the box prompts, while in terms of knowledge, the box prompts provide more information than the point prompts. Therefore, in our reasoning stage, we only use the box prompt branch for SAM. In the training stage, the point prompts of the CLIPS branch are still used to generate pseudo-truth boxes to supervise the geometric prompt boxes generated by GroundingDINO, so that the missing boxes of GroundingDINO can be supplemented by image-level information. In this embodiment of the present invention, we divide the point prompts generated by CLIPS into: core points, boundary points, and noise points. We remove noise points and cluster the remaining points into different groups to generate pseudo-truth boxes. The pseudocode of the "point to box" step is expressed as

[0058]

[0059] The semantic text adjuster is used to adjust the target text prompt, and the box prompt adjuster is used to adjust the box prompt. At the same time, the point generation branch and box generation branch of SAM are also changing dynamically. The above point-to-box strategy constructs box supervision, and the pixel-level supervision signal comes from the masks generated by the two SAMs. The box and mask losses are expressed as:

[0060]

[0061] Where Ci is the target class, is the matching loss of the box hint regulator part, It is a four-dimensional vector composed of the center coordinates of the detection box and its height and width relative to the image size, which respectively represent the pseudo-truth box generated by the point generation branch and the geometric hint box in the box generation branch. are the pixel values ​​of the predicted mask in the point generation branch and the box generation branch, respectively. α and β are the balance coefficients of focal loss and dice loss in mask loss. N is the number of training samples. L iou is the IOU loss, and ||.||1 is the L1 regularization loss. After supervised training with these two losses, the semantic text regulator is finally trained.

[0062] Finally, the image to be segmented is input into the semantic text adjuster TextTuner trained by the box supervision signal and the pixel-level supervision signal to adjust CLIPS and obtain the category semantics T that determines the existence of the image;

[0063] T=TextTuner●CLIPS(I,Classes);

[0064] Step S20: Generate region-level box hints. Under the guidance of category semantics, use the adjusted detection model GroundingDINO to detect the region where each category is located and output geometric box hints;

[0065] Step S20 includes the following sub-steps:

[0066] Step S21, extract semantic tokens with position information from SAM. The lightweight mask decoder in SAM implements mapping the image embedding and a set of prompt embeddings into segmentation maps with confidence scores. After each attention layer, the geometric position information of different types of prompt tokens provided is fully utilized by re-adding the entire original prompt token to the updated token. The geometric information is finally packaged in the form of "output token per task for each segmentation map", and the prompt segmentation map is finally dynamically generated by performing a dot product between the MLP weights predicted by the output token and the updated image embedding. In the above process, the output token has a "position-aware" property, and once the region-level box prompt is input into SAM, a region-level position-aware semantic token will be generated. In our framework, GroundingDINO is responsible for annotating the box, expressed as B={B i}, i = 1, .... N, N is the number of boxes. Accordingly, the semantic token extracted from SAM is represented as S = {S i}, i=1,....N.

[0067] Step S22: Box hint regulator training. A single image can capture comprehensive context from multiple objects belonging to hundreds of categories. Therefore, the image-level feature map can be conceptually viewed as a combination of region-level embeddings with different structures. Given an image I, the image-level image semantic map F I From CLIPS image features. After extracting the region-level semantic token S from SAM, a linear mapping operation is performed to project S to the region corresponding to F. I In the same feature space, the transformed token features can be expressed as F S What our box hint regulator does is to extract region-level features from image-level semantic feature maps, so we design it as a decoder module where F S plays the role of query and extracts region-related features from FI (keys and values) through cross attention. The cross attention between SAM and CLIPS can be expressed as

[0068]

[0069] Where KFI It is a key, V FI is the value, given by F I It is obtained by two separate linear projection operations. C is the number of channels in the feature space of the image encoder. Considering the broad semantics of the entire image, the regional feature F = {F i}, i=1,....N is extracted by integrating the position and semantic information of two different models.

[0070] Step S23, alignment supervision signal construction. The target text prompt is generated by CLIPS. Due to cross-domain mismatch noise, GroundingDINO will mistakenly label other regions as target text. In order to eliminate false detection and make it more amenable to CLIPS, we use the frozen CLIPS text encoder to easily capture the text embedding of a specific category, and then calculate the continuous similarity between the target text embedding T and the regional feature F to obtain the matching score, which is expressed as

[0071]

[0072] Among them, cos(.,.) represents cosine similarity; τ is a learnable temperature value. i is the matching score and also the attention weight for removing falsely detected bounding boxes. In the training phase, the box hint regulator is trained by minimizing the cross entropy loss between the matching score and the target text hint. In the test phase, different A i , through A i >0.5 for box screening, B={B j}, j=1, ....M, M≤N is the geometric box prompt output after adjustment.

[0073] Step S30: Generate pixel-level semantic segmentation map. Set the geometric box hint B = {B j}, j = 1, ....M, M ≤ N and semantic text prompts are input into SAM, and semantic labels are assigned pixel by pixel to the image to be segmented to achieve semantic segmentation.

[0074] Preferably, step S30 mainly inputs the box prompts automatically generated and optimized in the above steps into SAM for segmentation. The frozen SAM here is used as a segmentation tool to obtain a segmentation map with semantics.

[0075] Output = SAM (I, Prompt text ,Prompt box )

[0076] Among them, Prompt text and Prompt box It is an automatically generated text prompt and box prompt.box B generated in step S20 = {B j}, j=1, ....M, M≤N. Prompt text is the category semantics T of the image in step S10, and I is the image to be segmented.

[0077] Compared with the existing state-of-the-art, cue-based large-scale semantic segmentation methods, which perform fine recognition on hundreds of categories in the pixel-level LUSS task, we develop a training-free cascade pipeline that effectively exploits the extensive generalization knowledge of pre-trained visual grounding models (VFMs). Among them, CLIPS focuses on distinguishing category concepts and provides target semantic labels for GroundingDINO, while the generated box cues are subsequently used by SAM for semantic segmentation. However, the bottleneck of each VFM limits further performance improvement on the LUSS task. These limitations appear in the form of incorrect semantic and geometric cues, which are input into SAM, resulting in confusing semantic labels and inaccurately segmented shapes in the final output stage. Therefore, how to refine cues to enhance the adaptability of VFMs remains the research focus of this paper. Different from existing VFM refinement techniques that require fine-tuning pre-training parameters or use a small amount of annotated data for supervision, we propose trainable semantic context regulators and box regulators to utilize domain-specific knowledge from large-scale training samples to refine frozen VFMs unsupervised. The semantic context regulator is a trainable continuous vector that is used to regulate the recognition model CLIPS to improve its recognition accuracy for multiple categories. As a decoder, the box tuner promotes the interaction between image-level CLIPS and pixel-level SAM, improving the accuracy of the detection model Grounding DINO in localizing the semantic regions of objects. These two regulators help refine image-level text cues and region-level box cues while enhancing category-shape representations to meet the requirements of the LUSS task.

[0078] Example 2

[0079] Embodiment 2 of the present invention provides a terminal device corresponding to the above-mentioned embodiment 1. The terminal device may be a processing device for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the method of the above-mentioned embodiment.

[0080] The terminal device of this embodiment includes a memory, a processor, and a computer program stored in the memory; the processor executes the computer program in the memory to implement the steps of the method in the above-mentioned embodiment 1.

[0081] In some implementations, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0082] In some other implementations, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.

[0083] Example 3

[0084] Embodiment 3 of the present invention provides a computer-readable storage medium corresponding to the above embodiment 1, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method of the above embodiment 1 are implemented.

[0085] Computer readable storage media can be tangible devices that hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any combination thereof.

[0086] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0087] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0089] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0090] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A segmentation method adapted to large-scale pixel-level unsupervised semantic segmentation tasks, characterized in that: The following steps are involved: S1. Use the image to be segmented as the input of CLIPS and perform text-image matching to determine the category semantics of the image. S2. Under the guidance of the category semantics, the detection model is used to detect the area where each category is located to obtain a geometric box prompt; S3. Use the geometric box prompt as the input of SAM, assign semantic labels to the image to be segmented pixel by pixel, and obtain semantic segmentation results.

2. The segmentation method adapted to pixel-level large-scale unsupervised semantic segmentation tasks according to claim 1, characterized in that: In step S1, the CLIPS includes an image encoder and a text encoder connected in sequence; the image encoder takes a set of image patch embeddings as the Lth i+1 Input to the Transformer layer, with a learnable [CLS] token: y=Head * ([x N ]); where Ei is the [CLS] embedding feeding the Li+1 layer, [.,.] denotes stacking and concatenation operations, and * denotes frozen parameters. Indicates that the parameters of the Li layer network are frozen, and Head embeds the [CLS] of the last layer into x N The classification head projects into the probability distribution of the predicted class, where N is the number of Transformer layers.

3. The segmentation method adapted to pixel-level large-scale unsupervised semantic segmentation tasks according to claim 1, characterized in that: In step S2, the specific process of generating the geometric frame prompt includes: 1) Sort all the positive geometric points generated by CLIPS and select the middle point as the initial point; 2) Find all points within the radius of the initial point. If the number of points away from the initial point is less than Minpts, the initial point is marked as noise; otherwise, the initial point is marked as a core sample and a new cluster label is assigned; 3) Visit all neighbors of the initial point and assign cluster labels to the points that are not assigned to clusters; if they are core samples, visit their neighbors in turn, and so on; the cluster gradually increases until there are no core samples within the radius of the cluster; 4) Select another point that has not been visited and repeat steps 1) to 3) until all points are clustered into M clusters; 5) Traverse each cluster and find the upper left and lower right coordinates in each cluster as the pseudo-truth box generated by this cluster point, that is, obtain the geometric box prompt.

4. The segmentation method adapted to pixel-level large-scale unsupervised semantic segmentation tasks according to claim 1, characterized in that: In step S3, SAM includes a point generation branch and a box generation branch. The loss function expression of the box generation branch is: Among them, C i is the target class, is the matching loss of the box hint regulator, N is the number of training samples, It is a four-dimensional vector composed of the center coordinates of the detection box and its height and width relative to the image size. y represents the pseudo-truth box generated by the point generation branch. represents the geometric hint box in the pseudo-truth box generation branch, L iou is the IOU loss, ||.||1 is the L1 regularization loss, λ L1 is the weight coefficient; Loss of masks generated by point generation branch and box generation branch for: z, are the pixel values ​​of the predicted masks in the point generation branch and the box generation branch, respectively, and α and β are the balance coefficients of the focal loss and the dice loss in the mask loss.

5. The segmentation method adapted to pixel-level large-scale unsupervised semantic segmentation tasks according to claim 1, characterized in that: The semantic segmentation result Output is expressed as: Output=SAM(I,Prompttext,Promptbox); among them, Prompt text and Prompt box are automatically generated text prompts and geometric box prompts, and I is the image to be segmented.

6. A computer device comprising a memory, a processor and a computer program stored in the memory; characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program / instruction stored thereon; characterized in that: When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.

8. A computer program product comprising a computer program / instructions; characterized in that: When the computer program / instructions are executed by a processor, the steps of the method of any one of claims 1 to 5 are implemented.

Images