An open-vocabulary semantic segmentation method based on adaptive prototype caching at test time

Abstract

The application provides an open vocabulary semantic segmentation method based on adaptive prototype caching during testing. The method comprises obtaining an image to be segmented and an open vocabulary category set and loading a frozen model; a unified prediction unit containing visual features, category probability distribution and spatial support information is constructed using the frozen model; reliable prediction units are screened through confidence filtering and perturbation consistency checking, and class-level visual prototypes are obtained through classification and aggregation; the class-level visual prototypes are used to update an adaptive prototype cache initialized by text embedding; cache refinement scores are generated according to the cache prototypes and fused with original prediction scores to obtain pixel-level semantic segmentation results. By using stable and reliable visual evidence in the test image to supplement the static text prototype during the inference stage, the prediction accuracy and adaptation ability of the open vocabulary semantic segmentation in different model architectures and unseen category scenarios can be improved, and the method can be widely adapted to various computer vision tasks such as automatic driving and complex scene perception.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and open vocabulary semantic segmentation technology, and in particular to an open vocabulary semantic segmentation method based on test-time adaptive prototype caching. Background Technology

[0002] Open-vocabulary semantic segmentation aims to semantically label pixels in an input image based on arbitrary text categories. This enables the semantic segmentation model to not only identify known categories within the training label space but also to identify unseen semantic categories not encountered during the training phase. This technology has significant application value for tasks such as open-scene visual understanding, autonomous driving, robot perception, medical image analysis, remote sensing image interpretation, video surveillance, and human-computer interaction.

[0003] With the development of visual-language pre-trained models, open-vocabulary semantic segmentation methods based on text prototypes have become an important technical approach. These methods typically utilize a text encoder to convert category names or descriptions into text embeddings, using these embeddings as static semantic prototypes. Image features, mask features, object query features, or pixel-level features are then matched with the text prototypes based on similarity to complete open-vocabulary category prediction. Existing methods have improved the usability of visual-language priors in dense prediction tasks through pixel-level visual-text embeddings, post-mask classification, dense image-text matching, cue learning, lightweight adapters, or feature calibration.

[0004] Currently, most open-vocabulary semantic segmentation methods rely on pre-computed static text prototypes as semantic anchors. While static text prototypes are compact and transferable, they use the same image-independent semantic anchors for all test images, making it difficult to reflect differences in appearance, scale, context, local visual patterns, and image source within the current test image. In actual inference, the intermediate predictions output by the base model often already contain highly confident and semantically stable local regions or mask candidates. However, traditional static text prototype inference paradigms typically discard these reliable visual evidences directly after a single forward prediction, failing to organize them into a visual prototype that can be used to refine the prediction of the current image.

[0005] The drawbacks of one of the existing open-vocabulary semantic segmentation methods mentioned above include: while test-time adaptive techniques can improve model performance in scenarios with distributional offsets using test-phase data, most test-time adaptive methods require updating model parameters, performing backpropagation, or relying on image-level prediction hypotheses, making it difficult to directly adapt to the dense prediction scenarios in open-vocabulary semantic segmentation. Unlike image classification, open-vocabulary semantic segmentation generates a large number of spatially correlated local prediction units in a single inference step. These prediction units may contain noise, redundancy, boundary ambiguity, and pseudo-label errors. Indiscriminately including all prediction units in the cache or memory will lead to error accumulation and prototype contamination, ultimately affecting the final segmentation accuracy. Summary of the Invention

[0006] This invention provides an open-vocabulary semantic segmentation method based on test-time adaptive prototype caching, which utilizes reliable visual evidence during the testing phase to supplement static text prototypes, thereby improving the accuracy, stability, and architecture adaptability of open-vocabulary semantic segmentation results while freezing the basic open-vocabulary semantic segmentation model.

[0007] To achieve the above objectives, the present invention adopts the following technical solution.

[0008] An open-lexical semantic segmentation method based on test-time adaptive prototype caching includes:

[0009] Obtain the image to be segmented and the open vocabulary category set consisting of text categories, and load the frozen open vocabulary semantic segmentation model;

[0010] The image to be segmented and the open vocabulary category set are input into the open vocabulary semantic segmentation model to obtain multiple unified prediction units. Each unified prediction unit includes visual features, category probability distribution and spatial support information.

[0011] The unified prediction unit is subjected to reliability screening. The reliable prediction units that pass the reliability screening are grouped according to the prediction category. The reliable prediction units under each category are aggregated with confidence weight to obtain the class-level visual prototype of the corresponding category.

[0012] The adaptive prototype cache is updated using the class-level visual prototype. The visual features of the unified prediction unit are matched with the cached prototypes in the adaptive prototype cache to obtain a cache refinement score. The cache refinement score is fused with the original prediction score output by the open vocabulary semantic segmentation model to obtain a refined category probability distribution. The open vocabulary semantic segmentation result of the image to be segmented is output according to the refined category probability distribution.

[0013] Preferably, the step of acquiring the image to be segmented and the open vocabulary category set composed of text categories, and loading the frozen open vocabulary semantic segmentation model, includes:

[0014] Obtain the image to be segmented I and the open vocabulary category set C, and load the frozen open vocabulary semantic segmentation model. The open vocabulary category set C consists of the text category names input by the user and / or the category vocabulary list preset by the target application scenario. The frozen open vocabulary semantic segmentation model includes an image encoding part, a text encoding part, and a query-based or pixel-based segmentation prediction part, and the model parameters remain unchanged. The frozen open vocabulary semantic segmentation model outputs the original prediction score and visual features of the image to be segmented I.

[0015] The open-vocabulary semantic segmentation model and adaptive prototype cache structure include a frozen open-vocabulary semantic segmentation model, a unified prediction unit construction module, a reliable prediction unit selection and class-level visual prototype construction module, an adaptive prototype cache update module, and a prototype-guided segmentation prediction refinement module. The frozen open-vocabulary semantic segmentation model provides the original prediction scores and visual features, and its model parameters remain unchanged throughout all processing stages.

[0016] Preferably, the step of inputting the image to be segmented and the open vocabulary category set into the open vocabulary semantic segmentation model yields multiple unified prediction units. Each unified prediction unit includes visual features, category probability distribution, and spatial support information, including:

[0017] The image to be segmented I is input into the image encoding part of the frozen open vocabulary semantic segmentation model to extract the image visual features of the image to be segmented I; the open vocabulary category set C is input into the text encoding part of the frozen open vocabulary semantic segmentation model to obtain the text embeddings corresponding to each category; the query-based or pixel-based segmentation prediction part of the frozen open vocabulary semantic segmentation model performs forward inference based on the image visual features and text embeddings, and outputs the original prediction score and the visual features corresponding to the prediction position or prediction object;

[0018] For the i-th unit to be constructed, obtain the original prediction score or the similarity score between the visual features and the text embedding of the unit to be constructed. The class probability distribution is obtained by using a normalized exponential function:

[0019]

[0020] Visual features, category probability distributions, and spatial support information are combined to form a unified set of prediction units. :

[0021]

[0022] Including A unified prediction unit Represents the visual features of the i-th unified prediction unit. Indicates the first A unified prediction unit in an open vocabulary category set Class probability distribution on, Indicates the first Spatial support information for a unified prediction unit.

[0023] Preferably, the reliability screening of the unified prediction unit involves grouping the reliable prediction units that pass the reliability screening according to prediction categories, and performing confidence-weighted aggregation on the reliable prediction units under each category to obtain the class-level visual prototype for the corresponding category, including:

[0024] For the Visual features of a unified prediction unit Normalize:

[0025]

[0026] Determine the predicted category based on the category probability distribution. and prediction confidence :

[0027]

[0028] in, Indicates the first Each unified prediction unit belongs to the category The probability of;

[0029] Prediction confidence Compared with the preset confidence threshold When comparing, Not greater than the preset confidence threshold When, the corresponding unified prediction unit is excluded; when Greater than the preset confidence threshold At that time, the perturbation consistency check is continued for the unified prediction unit;

[0030] Adding Gaussian noise to the normalized visual features and then renormalizing them yields perturbed visual features. :

[0031]

[0032] in It is Gaussian perturbation noise. The larger the value, the greater the degree of perturbation applied to the original visual features, and the more it will perturb the visual features. Input the original classification header or text matching operation of the frozen model. The perturbation prediction score is obtained. Perturbation category probability distribution and disturbance prediction categories :

[0033]

[0034] The reliability label is determined based on the confidence filtering results and the consistency of the categories before and after the perturbation:

[0035]

[0036] in, Indicates an indicator function, when When, the corresponding unified prediction unit is determined as a reliable prediction unit; when When this happens, the corresponding unified prediction unit will be excluded;

[0037] Reliable prediction units are grouped according to prediction categories, for any category in the open vocabulary category set. Establish a reliable prediction unit index set :

[0038]

[0039] when When not empty, the category is determined based on each reliable prediction unit. The predicted probabilities are aggregated using confidence weighting to obtain the category. Class-level visual prototype:

[0040]

[0041] in, Indicates the first Category in the next reasoning step The corresponding class-level visual prototype.

[0042] Preferably, the step of updating the adaptive prototype cache using the class-level visual prototype includes:

[0043] Initialize the adaptive prototype cache based on the text embeddings corresponding to the open vocabulary category set, and the categories. Initial cache prototype for:

[0044]

[0045] in, Indicates category Text embedding;

[0046] When category The corresponding reliable prediction unit index set When not empty, utilize class-level visual prototypes. The historical cache prototype for this category is updated using an exponential moving average:

[0047]

[0048] in, This indicates the updated cache prototype. This represents the prototype of the historical cache. Indicates the momentum coefficient. ;

[0049] When category The corresponding reliable prediction unit index set When empty, do not update the cached prototype of this category using the current test image, and maintain... .

[0050] Preferably, the step of matching the visual features of the unified prediction unit with the cached prototypes in the adaptive prototype cache to obtain a cache refinement score, fusing the cache refinement score with the original prediction score output by the open-vocabulary semantic segmentation model to obtain a refined category probability distribution, and outputting the open-vocabulary semantic segmentation result of the image to be segmented based on the refined category probability distribution includes:

[0051] For the i-th unified prediction unit and category c, calculate its normalized visual features. With the updated cache prototype Cosine similarity between :

[0052]

[0053] The cosine similarity values ​​corresponding to each category are used to construct the prototype similarity vector. and utilize temperature coefficient For prototype similarity vectors Perform a normalized exponential transformation to obtain the cache refinement score. :

[0054]

[0055] Among them, temperature coefficient Used to control the sharpness of the prototype matching distribution;

[0056] Refine the cache score The original prediction score compared to the output of the frozen open-vocabulary semantic segmentation model To merge:

[0057]

[0058] in, Indicates the fusion weight. This represents the predicted score after fusion. This represents the refined category probability distribution;

[0059] The refined category probability distribution and corresponding spatial support information are returned to the original mask aggregation and post-processing process of the frozen open-vocabulary semantic segmentation model. For query-based segmentation models, the mask aggregation and post-processing operations are used; for pixel-based segmentation models, the dense prediction post-processing operation is used. The frozen open-vocabulary semantic segmentation model outputs the pixel-level open-vocabulary semantic segmentation result of the image to be segmented. The pixel-level open-vocabulary semantic segmentation result includes the segmented region and the corresponding category label.

[0060] An open-vocabulary semantic segmentation apparatus based on test-time adaptive prototype caching includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the open-vocabulary semantic segmentation method based on test-time adaptive prototype caching.

[0061] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the open-vocabulary semantic segmentation method based on test-time adaptive prototype caching.

[0062] As can be seen from the technical solutions provided by the embodiments of the present invention described above, the present invention constructs mask candidates, object queries, or dense pixel predictions generated by different open-vocabulary semantic segmentation architectures into a unified prediction unit, and sequentially performs reliable prediction unit screening and class-level visual prototype construction, adaptive prototype cache update, and prototype-guided segmentation prediction refinement, thereby achieving lightweight enhancement of the prediction results of the frozen base model. The present invention requires no additional training, fine-tuning, backpropagation, or structural modification, and can be used as a plug-and-play inference stage refinement module in query-based and pixel-based open-vocabulary semantic segmentation models.

[0063] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0064] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 A flowchart of an open vocabulary semantic segmentation method based on test-time adaptive prototype caching provided in an embodiment of the present invention;

[0066] Figure 2 This is a structural diagram of an open vocabulary semantic segmentation model and adaptive prototype caching provided in an embodiment of the present invention;

[0067] Figure 3 A flowchart for constructing a unified prediction unit is provided in an embodiment of the present invention;

[0068] Figure 4 This is a flowchart of a reliable prediction unit selection and class-level visual prototype construction method provided in an embodiment of the present invention. Detailed Implementation

[0069] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0070] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0071] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0072] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0073] To address the problem that existing open-vocabulary semantic segmentation methods rely on image-independent static text prototypes and struggle to utilize reliable visual evidence from current test images, this invention transforms the test-time refinement process of open-vocabulary semantic segmentation into a problem of constructing a reliable visual prototype cache and fusing prototype-guided prediction. This invention does not retrain the base model, nor does it modify the encoder, decoder, text classifier, or post-processing modules within the base model. Instead, during the inference phase, it selects stable and reliable visual evidence from the prediction units output by the base model and organizes it into class-level adaptive visual prototypes to supplement the original text prototype inference.

[0074] A flowchart of an open-vocabulary semantic segmentation method based on test-time adaptive prototype caching provided by an embodiment of the present invention is shown below. Figure 1 As shown, it includes the following steps:

[0075] Step S1: Obtain the image to be segmented and the open vocabulary category set consisting of text categories, and load the frozen open vocabulary semantic segmentation model.

[0076] Obtain the image to be segmented, I, and the open vocabulary category set C, and load the frozen open vocabulary semantic segmentation model. The open vocabulary category set C can consist of text category names input by the user, or it can consist of a category vocabulary preset for the target application scenario, with the number of categories being [number missing]. .

[0077] The open vocabulary category set C can simultaneously include both seen categories that appeared during the model training phase and unseen categories that did not appear during the training phase, i.e. The frozen open-vocabulary semantic segmentation model includes an image encoding part, a text encoding part, and a query-based or pixel-based segmentation prediction part, and the model parameters remain unchanged during the execution of the method of the present invention.

[0078] The overall structure of an open-vocabulary semantic segmentation model and adaptive prototype cache provided in this embodiment of the invention is as follows: Figure 2 As shown, the architecture includes a frozen open-vocabulary semantic segmentation model, a unified prediction unit construction module, a reliable prediction unit selection and class-level visual prototype construction module, an adaptive prototype cache update module, and a prototype-guided segmentation prediction refinement module. The frozen open-vocabulary semantic segmentation model provides the raw prediction scores and visual features, and its model parameters remain unchanged throughout all processing stages; the remaining modules are all executed in a forward computation manner during testing, without training, fine-tuning, or backpropagation.

[0079] Step S2: Input the image to be segmented and the open vocabulary category set into the frozen open vocabulary semantic segmentation model to obtain multiple unified prediction units. Each unified prediction unit includes visual features, category probability distribution, and spatial support information. The specific process of this step is as follows: Figure 3 As shown, it includes the following sub-steps:

[0080] Step S2-1: Input the image to be segmented I into the image encoding part of the frozen open-vocabulary semantic segmentation model to extract the visual features of image I; input the open-vocabulary category set C into the text encoding part of the frozen open-vocabulary semantic segmentation model to obtain the text embeddings corresponding to each category. The query-based or pixel-based segmentation prediction part of the frozen open-vocabulary semantic segmentation model performs forward inference based on the image visual features and text embeddings, outputting the original prediction score and the visual features corresponding to the prediction position or prediction object.

[0081] Step S2-2: Determine spatial support information based on the architecture of the frozen open-vocabulary semantic segmentation model. For query-based segmentation, each mask candidate or object query is treated as a unit to be constructed, the binary mask of the unit to be constructed is used as spatial support information, and the query-level features are used as visual features. For pixel-based segmentation, each spatial location in the dense feature map is treated as a unit to be constructed, the corresponding pixel coordinates or feature map location is used as spatial support information, and the pixel-level features corresponding to that location are used as visual features.

[0082] Step S2-3: For the i-th unit to be constructed, obtain the original prediction score or the similarity score between the visual features and the text embedding of the unit to be constructed. The class probability distribution is obtained by using a normalized exponential function:

[0083]

[0084] Step S2-4: Combine visual features, category probability distributions, and spatial support information to form a unified set of prediction units:

[0085]

[0086] in, Represents the visual features of the i-th unified prediction unit. Indicates the first A unified prediction unit in an open vocabulary category set Class probability distribution on, Indicates the first Spatial support information for a unified prediction unit This indicates the number of unified prediction units. Through the above construction method, mask candidates, object queries, or dense spatial locations generated by different model architectures are transformed into unified prediction units with a consistent data structure.

[0087] Step S3: Perform reliability screening on all unified prediction units. The reliability screening includes confidence filtering based on category probability distribution and perturbation consistency verification based on the consistency of predicted categories before and after feature perturbation. Then, the reliable prediction units that have passed the reliability screening are grouped according to the prediction category, and the reliable prediction units under each category are aggregated with confidence weight to obtain the class-level visual prototype of the corresponding category.

[0088] The specific process of step S3 is as follows: Figure 4 As shown, step S3 includes the following sub-steps:

[0089] Step S3-1: For the first The visual features of each unified prediction unit are normalized:

[0090]

[0091] Determine the predicted category and prediction confidence level based on the category probability distribution:

[0092]

[0093] in, Indicates the first Each unified prediction unit belongs to the category The probability of.

[0094] Step S3-2: Calculate the prediction confidence level Compared with the preset confidence threshold Comparison. When Not greater than the preset confidence threshold When, the corresponding unified prediction unit is excluded; when Greater than the preset confidence threshold At that time, the perturbation consistency check is continued for the unified prediction unit.

[0095] Step S3-3: Add Gaussian noise to the normalized visual features and re-normalize to obtain the perturbed visual features:

[0096]

[0097] Perturbation of visual features Input the original classification header or text matching operation of the frozen model. This yields the perturbation prediction score, the perturbation category probability distribution, and the perturbation prediction category:

[0098]

[0099] Step S3-4: Determine the reliability label based on the confidence filtering results and the consistency of the categories before and after the disturbance.

[0100]

[0101] in, Indicates an indicator function. When When, the corresponding unified prediction unit is determined as a reliable prediction unit; when When this happens, the corresponding unified prediction unit will be excluded and will not be used for class-level visual prototype construction and adaptive prototype cache updates.

[0102] Step S3-5: Group reliable prediction units according to prediction category. For any category in the open vocabulary category set... Establish a reliable prediction unit index set:

[0103]

[0104] when When not empty, the category is determined based on each reliable prediction unit. The predicted probabilities are aggregated using confidence weighting to obtain the category. Class-level visual prototype:

[0105]

[0106] in, Indicates the first Category in the next reasoning step The corresponding class-level visual prototype. Through confidence-weighted aggregation, reliable predictive units with strong category evidence contribute significantly to the class-level visual prototype, and multiple local visual observations of the same category are combined into a class-level representation.

[0107] Step S4: Update the adaptive prototype cache using the class-level visual prototype. The adaptive prototype cache stores cached prototypes corresponding to each open vocabulary category. This includes the following sub-steps:

[0108] Step S4-1: Initialize the adaptive prototype cache based on the text embeddings corresponding to the open vocabulary category set. Categories The initial cache prototype was:

[0109]

[0110] in, Indicates category Text embeddings are used. By initializing with normalized text embeddings, the open-vocabulary semantic priors of the frozen model are preserved when reliable visual evidence is not yet available.

[0111] Step S4-2: When the category The corresponding reliable prediction unit index set If not empty, use the class-level visual prototype obtained in step S3-5. The historical cache prototype for this category is updated using an exponential moving average:

[0112]

[0113] in, This indicates the updated cache prototype. This represents the prototype of the historical cache. Indicates the momentum coefficient. .

[0114] Step S4-3: When the category The corresponding reliable prediction unit index set When empty, do not update the cached prototype of this category using the current test image, and maintain... This avoids polluting the adaptive prototype cache with low-quality predictions or sparse pseudo-labels.

[0115] Step S5: Match the visual features of the current unified prediction unit with the cached prototypes in the updated adaptive prototype cache to obtain a cache refinement score. Then, fuse the cache refinement score with the original prediction score output by the frozen open-vocabulary semantic segmentation model to obtain a refined category probability distribution. Based on the refined category probability distribution, output the pixel-level open-vocabulary semantic segmentation result of the image to be segmented, including the following sub-steps: Prototype-guided segmentation prediction refinement.

[0116] Step S5-1: For the i-th unified prediction unit and category c, calculate the cosine similarity between its normalized visual features and the updated cached prototype:

[0117]

[0118] Step S5-2: Combine the cosine similarities of each category into a prototype similarity vector. and utilize temperature coefficient Perform a normalized exponential transformation to obtain the cache refinement score:

[0119]

[0120] Among them, temperature coefficient Used to control the sharpness of the prototype matching distribution.

[0121] Step S5-3: Fuse the cached refined score with the original prediction score output by the frozen open-vocabulary semantic segmentation model:

[0122]

[0123] in, Indicates the fusion weight. This represents the predicted score after fusion. This represents the refined category probability distribution.

[0124] Step S5-4: Return the refined category probability distribution and corresponding spatial support information to the original segmentation post-processing workflow of the frozen model. For query-based segmentation models, use their mask aggregation and post-processing operations; for pixel-based segmentation models, use their dense prediction post-processing operations to output pixel-level open-vocabulary semantic segmentation results corresponding to the image to be segmented, including the segmented regions and corresponding category labels.

[0125] The above refinement process only introduces cached refinement signals into the original predicted scores, without replacing the original semantic priors and deployment process of the frozen model.

[0126] In one specific implementation, the confidence threshold The settings can be configured based on the application scenario, the reliability of the base model, or the performance of the validation set; Gaussian noise standard deviation. It can be set according to the visual feature dimensions and disturbance sensitivity; momentum coefficient It can be used to control the fusion ratio of historical prototypes and current image visual evidence; temperature coefficient T and fusion weights. This can be used to control the influence of the buffer refinement signal on the final prediction. Those skilled in the art will understand that the above parameters are not the only possible values ​​for this invention, and can be adjusted according to the actual deployment environment.

[0127] In one specific implementation, the adaptive prototype cache can be built and used only within a single test image, or it can retain historical prototypes across images during continuous inference. When dealing with video, continuous image streams, or multiple test images in the same deployment environment, the adaptive prototype cache can be continuously updated with each inference step to accumulate more stable class-level visual evidence; when dealing with one-off image inference, the adaptive prototype cache can also be released after the current image processing is completed to ensure that the inference process is independent.

[0128] In one specific implementation, the spatial support information The visual features can be binary masks, pixel coordinates, feature map positions, candidate regions, superpixel regions, or other information that can represent the image range corresponding to the prediction unit. The visual features can be query features, masked pooling features, pixel-level features, region features, or features derived from intermediate layers of the base model. The category probability distribution can be obtained by normalizing text prototype similarity, classification head output, masked classification results, or dense prediction results.

[0129] The present invention also provides an open vocabulary semantic segmentation apparatus based on test-time adaptive prototype caching, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned open vocabulary semantic segmentation method based on test-time adaptive prototype caching.

[0130] A computer-readable storage medium storing a computer program, the computer program being executed by a processor using the above-described open-vocabulary semantic segmentation method based on test-time adaptive prototype caching.

[0131] The beneficial effects of the method of the present invention are as follows:

[0132] First, by constructing a unified prediction unit, query-based models, pixel-based models, and other open lexical semantic segmentation architectures are transformed into a consistent inference stage refinement interface, which improves the method's architectural adaptability.

[0133] Second, by using confidence filtering and perturbation consistency verification, only stable and reliable prediction units are selected to build class-level visual prototypes, which reduces the pollution of the cache by noise pseudo-labels and boundary ambiguity.

[0134] Third, by text embedding initialization and momentum update, reliable visual evidence from the current test image is absorbed while preserving the semantic prior of open-vocabulary text;

[0135] Fourth, by refining segmentation prediction through prototype-guided methods, the cached refinement score is used as a supplementary signal to the original prediction score, enabling adaptive prototype caching to improve local ambiguity, foreground-background confusion, and semantic category shift issues.

[0136] Fifth, this invention does not require additional training, fine-tuning, backpropagation, or modification of the basic model structure, making it suitable for application in already deployed open-ended semantic segmentation systems.

[0137] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0138] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0139] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0140] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An open-vocabulary semantic segmentation method based on test-time adaptive prototype caching, characterized in that, include: Obtain the image to be segmented and the open vocabulary category set consisting of text categories, and load the frozen open vocabulary semantic segmentation model; The image to be segmented and the open vocabulary category set are input into the open vocabulary semantic segmentation model to obtain multiple unified prediction units. Each unified prediction unit includes visual features, category probability distribution and spatial support information. The unified prediction unit is subjected to reliability screening. The reliable prediction units that pass the reliability screening are grouped according to the prediction category. The reliable prediction units under each category are aggregated with confidence weight to obtain the class-level visual prototype of the corresponding category. The adaptive prototype cache is updated using the class-level visual prototype. The visual features of the unified prediction unit are matched with the cached prototypes in the adaptive prototype cache to obtain a cache refinement score. The cache refinement score is fused with the original prediction score output by the open vocabulary semantic segmentation model to obtain a refined category probability distribution. The open vocabulary semantic segmentation result of the image to be segmented is output according to the refined category probability distribution.

2. The method according to claim 1, characterized in that, The process of acquiring the image to be segmented and the open vocabulary category set consisting of text categories, and loading the frozen open vocabulary semantic segmentation model, includes: Obtain the image to be segmented I and the open vocabulary category set C, and load the frozen open vocabulary semantic segmentation model. The open vocabulary category set C consists of the text category names input by the user and / or the category vocabulary list preset by the target application scenario. The frozen open vocabulary semantic segmentation model includes an image encoding part, a text encoding part, and a query-based or pixel-based segmentation prediction part, and the model parameters remain unchanged. The frozen open vocabulary semantic segmentation model outputs the original prediction score and visual features of the image to be segmented I. The open-vocabulary semantic segmentation model and adaptive prototype cache structure include a frozen open-vocabulary semantic segmentation model, a unified prediction unit construction module, a reliable prediction unit selection and class-level visual prototype construction module, an adaptive prototype cache update module, and a prototype-guided segmentation prediction refinement module. The frozen open-vocabulary semantic segmentation model provides the original prediction scores and visual features, and its model parameters remain unchanged throughout all processing stages.

3. The method according to claim 2, characterized in that, The process involves inputting the image to be segmented and the open vocabulary category set into the open vocabulary semantic segmentation model to obtain multiple unified prediction units. Each unified prediction unit includes visual features, category probability distribution, and spatial support information, including: The image to be segmented I is input into the image encoding part of the frozen open vocabulary semantic segmentation model to extract the image visual features of the image to be segmented I; the open vocabulary category set C is input into the text encoding part of the frozen open vocabulary semantic segmentation model to obtain the text embeddings corresponding to each category; the query-based or pixel-based segmentation prediction part of the frozen open vocabulary semantic segmentation model performs forward inference based on the image visual features and text embeddings, and outputs the original prediction score and the visual features corresponding to the prediction position or prediction object; For the i-th unit to be constructed, obtain the original prediction score or the similarity score between the visual features and the text embedding of the unit to be constructed. The class probability distribution is obtained by using a normalized exponential function: （1） Visual features, category probability distributions, and spatial support information are combined to form a unified set of prediction units. : （2） Including A unified prediction unit Represents the visual features of the i-th unified prediction unit. Indicates the first A unified prediction unit in an open vocabulary category set Class probability distribution on, Indicates the first Spatial support information for a unified prediction unit.

4. The method according to claim 3, characterized in that, The aforementioned reliability screening of the unified prediction unit involves grouping the reliable prediction units that pass the reliability screening according to prediction categories, and performing confidence-weighted aggregation on the reliable prediction units under each category to obtain the corresponding category-level visual prototype, including: For the first Visual features of a unified prediction unit Normalize: （3） Determine the predicted category based on the category probability distribution. and prediction confidence : （4） in, Indicates the first Each unified prediction unit belongs to the category The probability of; Prediction confidence Compared with the preset confidence threshold When comparing, Not greater than the preset confidence threshold When, the corresponding unified prediction unit is excluded; when Greater than the preset confidence threshold At that time, the perturbation consistency check is continued for the unified prediction unit; Adding Gaussian noise to the normalized visual features and then renormalizing them yields perturbed visual features. : （5） in It is Gaussian perturbation noise. The larger the value, the greater the degree of perturbation applied to the original visual features, and the more it will perturb the visual features. Input the original classification header or text matching operation of the frozen model. The perturbation prediction score is obtained. Perturbation category probability distribution and disturbance prediction categories : （6） The reliability label is determined based on the confidence filtering results and the consistency of the categories before and after the perturbation: （7） in, Indicates an indicator function, when When, the corresponding unified prediction unit is determined as a reliable prediction unit; when When this happens, the corresponding unified prediction unit will be excluded; Reliable prediction units are grouped according to prediction categories, for any category in the open vocabulary category set. Establish a reliable prediction unit index set : （8） when When not empty, the category is determined based on each reliable prediction unit. The predicted probabilities are aggregated using confidence weighting to obtain the category. Class-level visual prototype: （9） in, Indicates the first Category in the next reasoning step The corresponding class-level visual prototype.

5. The method according to claim 4, characterized in that, The method of updating the adaptive prototype cache using the class-level visual prototype includes: Initialize the adaptive prototype cache based on the text embeddings corresponding to the open vocabulary category set, and the categories. Initial cache prototype for: （10） in, Indicates category Text embedding; When category The corresponding reliable prediction unit index set When not empty, utilize class-level visual prototypes. The historical cache prototype for this category is updated using an exponential moving average: （11） in, This indicates the updated cache prototype. This represents the prototype of the historical cache. Indicates the momentum coefficient. ; When category The corresponding reliable prediction unit index set When empty, do not update the cached prototype of this category using the current test image, and maintain... .

6. The method according to claim 5, characterized in that, The process involves matching the visual features of the unified prediction unit with the cached prototypes in the adaptive prototype cache to obtain a cache refinement score, fusing the cache refinement score with the original prediction score output by the open-vocabulary semantic segmentation model to obtain a refined category probability distribution, and outputting the open-vocabulary semantic segmentation result of the image to be segmented based on the refined category probability distribution. This includes: For the i-th unified prediction unit and category c, calculate its normalized visual features. With the updated cache prototype Cosine similarity between : （12） The cosine similarity values ​​corresponding to each category are used to construct the prototype similarity vector. and utilize temperature coefficient For prototype similarity vectors Perform a normalized exponential transformation to obtain the cache refinement score. : （13） Among them, temperature coefficient Used to control the sharpness of the prototype matching distribution; Refine the cache score The original prediction score compared to the output of the frozen open-vocabulary semantic segmentation model To merge: （14） in, Indicates the fusion weight. This represents the predicted score after fusion. This represents the refined category probability distribution; The refined category probability distribution and corresponding spatial support information are returned to the original mask aggregation and post-processing process of the frozen open-vocabulary semantic segmentation model. For query-based segmentation models, the mask aggregation and post-processing operations are used; for pixel-based segmentation models, the dense prediction post-processing operation is used. The frozen open-vocabulary semantic segmentation model outputs the pixel-level open-vocabulary semantic segmentation result of the image to be segmented. The pixel-level open-vocabulary semantic segmentation result includes the segmented region and the corresponding category label.

7. An open-lexical semantic segmentation device based on test-time adaptive prototype caching, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the open-vocabulary semantic segmentation method based on test-time adaptive prototype caching as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the open-vocabulary semantic segmentation method based on test-time adaptive prototype caching as described in any one of claims 1-6.

Images