Open-vocabulary segmentation method and system with multi-modal model representation optimization

Abstract

The application provides an open vocabulary segmentation method and system for multi-modal model representation optimization, and belongs to the technical field of computer vision. Image data to be segmented is acquired; a pre-trained multi-modal model is used to process the acquired image to obtain a segmentation result. The application better optimizes visual-text representation in a multi-modal task, effectively aligns the same visual-text representation space, proposes a mask-sensitive loss to constrain the classification score and mask quality to be consistent in the parameter fine-tuning process, thereby giving the visual encoder local perception ability and improving the effect of the model in the fine-grained downstream task, introduces the original pre-training feature as a representation compensation to ensure the zero-shot ability of the pre-training visual-language model in the optimization process, and interacts the text representation and the visual representation, so that the text representation can be adaptively enhanced for different input images, and the alignment property of the visual-text in the open vocabulary segmentation can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to an open vocabulary segmentation method and system for multimodal model representation optimization. Background Technology

[0002] Semantic segmentation is one of the most widely studied topics in computer vision. With the rapid development of deep learning technology, semantic segmentation techniques have made significant progress. However, traditional semantic segmentation models require data of the same category and style for both training and testing. This means that the trained model can only segment predefined categories within a closed vocabulary, and the number of these categories is far less than the number of categories humans need to describe the real world. Therefore, this limitation greatly reduces the value of traditional semantic segmentation techniques in practical applications. To address this problem, open-vocabulary segmentation tasks have emerged. Open-vocabulary segmentation breaks through the limitation of segmentation categories, enabling segmentation based on any given category, thus significantly expanding the application scope of segmentation tasks and improving their practicality in various real-world scenarios.

[0003] In open-vocabulary segmentation research, early methods attempted to achieve this goal by learning a unified visual-text conceptual space. These methods leverage visual context to guide the generation of textual features or utilize vocabularies such as WordNet to generate pixel-level visual features. However, due to the lack of a large-scale visual-text aligned representation space, these methods suffer from bottlenecks when handling unknown categories, making it difficult to achieve satisfactory results. For example, how to effectively fuse visual and textual information to maintain high accuracy in fine-grained image regions remains a pressing issue. Furthermore, while ensuring strong generalization ability in open-vocabulary scenarios, avoiding overfitting is also a crucial challenge.

[0004] With the success of large-scale visual-text pre-trained models (such as CLIP and ALIGN), open vocabulary segmentation models have made significant progress by leveraging the powerful visual-text alignment properties of these pre-trained models. Recent open vocabulary segmentation models typically follow a "segmentation-classification decoupling" paradigm, decoupling the open vocabulary segmentation task into two parts: prior mask generation and prior mask classification. First, these methods generate a series of prior masks using a class-independent segmenter, and then use CLIP or ALIGN to classify the regions covered by the prior masks in the image. Specifically, in the first stage, open vocabulary segmentation models typically employ Hungarian matching to match the prior masks with the annotations, training and optimizing only the matched masks. This method effectively avoids overfitting of the prior masks to the training classes, maintaining their generalization ability. In the second stage of prior mask classification, some methods process at the image level by erasing the background region in the prior mask and cropping a sub-image containing only the foreground region from the original image for classification; other methods process at the feature level by extracting features containing only the foreground region through masked average pooling or masked attention mechanisms for classification.

[0005] To further improve the performance of open-vocabulary segmentation, a series of works, represented by OpenSeg, use additional image-title pairs to expand the training data, resulting in more robust segmentation performance. Works such as FreeSeg unify semantic, instance, and panoptic segmentation tasks and perform fusion training. Works such as ODISE utilize a powerful text-to-image diffusion model (StableDiffusion) to obtain a stronger visual-text feature space. Although the above methods improve the performance of open-vocabulary segmentation to some extent, they typically freeze the pre-trained model parameters during training to maintain the aligned feature space of the pre-trained visual-language model. However, since pre-trained large visual-language models are usually trained using image-title pairs and lack local perception capabilities, pixel-level segmentation in current tasks remains very difficult.

[0006] In summary, existing methods ensure zero-shot capability by freezing the parameters of pre-trained models. However, since pre-trained models are typically designed for classification tasks and lack local perception capabilities, directly applying pre-trained visual-language models to fine-grained segmentation tasks can lead to significant domain discrepancies. The performance upper bound of existing methods remains limited by the pre-trained visual-text model, making it difficult to overcome these limitations and resulting in poor performance when handling complex scenes and diverse categories. Summary of the Invention

[0007] The purpose of this invention is to provide an open vocabulary segmentation method and system for multimodal model representation optimization, so as to solve at least one of the technical problems existing in the background art.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides an open vocabulary segmentation method for multimodal model representation optimization, comprising:

[0010] Obtain the image data to be segmented;

[0011] The acquired images are processed using a pre-trained multimodal model to obtain segmentation results. The training of the multimodal model includes: in the feature extraction stage, feature encoding is performed on the image and text respectively, where visual features are fed into a prior mask generator to generate class-independent prior masks, and text features are used for classification; in the feature interaction stage, visual and text features are enhanced based on content-dependent representation transfer; based on mask-sensitive loss and representation compensation loss, local perceptual capabilities are imparted to the visual encoder through parameter fine-tuning, while replaying the original visual features to avoid overfitting; end-to-end training is used to co-optimize visual-text representations to obtain a better visual-text alignment space; until the model test results meet expectations or the training iterations are reached.

[0012] Furthermore, features are extracted from the input image using a pre-trained CLIP visual encoder, and the output of each stage of the visual encoder is represented as F = {F...} i}, i = [0, 1, 2, 3]; MaskFormer is used as the prior mask generator; Hungarian matching is used to optimize the subset of the prior mask that has been matched during training; gradient propagation from the visual encoder to the prior mask generator is terminated to avoid overfitting of the visual encoder on the training categories.

[0013] Furthermore, a mask-aware loss L is introduced. ma To fine-tune the visual encoder using mask-sensitive features, enabling it to have local perceptual capabilities, including: L ma The goal is to assign high classification scores to high-quality prior masks and low classification scores to low-quality prior masks. Specifically, the intersection-union (IoU) score between the prior mask and the labels is calculated and aligned with the classification scores. Assuming there are k classes in the ground truth, k binary graphs can be generated from the labels, and their IoU score with the prior mask is calculated and denoted as S. IoU .

[0014] Furthermore, the maximum value of the classification score and S IoU There are differences between the maximum values: the maximum value of the classification score usually tends to be close to 1, while the maximum value of S...IoU The maximum value fluctuates between 0.75 and 0.99, and this inconsistency increases the classification score and S. IoU Alignment difficulty between them; therefore, for S IoU The following normalization was introduced:

[0015]

[0016] In addition, L ma Select k categories from the classification scores that exist in the labels, denoted as S. pre and use the SmoothL1 function to combine it with Alignment.

[0017] Furthermore, the visual representation compensation strategy aims to enhance the zero-shot capability of the fine-tuned visual encoder by replaying the original features of the CLIP during the training phase as representation compensation; the original visual features generated during training are used as compensation features by the CLIP visual encoder with one parameter frozen, and these are incorporated into the fine-tuned visual encoder; wherein, the representation compensation loss L RC To constrain the final layer output of CLIP-V* and the final layer output F of the fine-tuned visual encoder 3 To maintain consistency and enhance the zero-sample capability of the visual encoder.

[0018] Furthermore, directly and F 3 Pixel-by-pixel alignment leads to the loss of regional differences. Therefore, multi-scale average pooling (AvgPooling) is used to generate multi-scale features, and the consistency of the pooled features is constrained.

[0019] For any feature f, the average pooling operation with a grid size of k×k is expressed as:

[0020]

[0021] exist and F 3 Based on this, multi-scale average pooling with a grid size of K = {1, 2, 4} is used to transform it into a grid of {1×1, 2×2, 4×4}, denoted as and F p ,Right now: F p =AvgPooling(F 3 ,K); where L RC SmoothL1 was used to minimize and F p Differences:

[0022] Furthermore, the content-dependent representation transfer module includes a series of Transformer layers that undergo cross-attention with visual features; wherein, the final layer features F of the visual encoder are transferred... 3 And the text representation T is used as input for content-dependent representation transfer. First, F 3 The reordering operation is performed in the spatial dimension, followed by processing T and F using n consecutive Transformer layers. 3 Simultaneously, residual connections are introduced, and this process is expressed as:

[0023] T i+1 =TransLayer i (T i ,F 3 )+T i i = 1, 2, ... L

[0024] The output of content-dependent representation transfer is the optimized text representation, denoted as . The Transformer layer is represented by the following formula:

[0025]

[0026] Where Que(·), Key(·), and Val(·) represent linear projections, and d is the dimension of the input vector.

[0027] Secondly, the present invention provides an open vocabulary segmentation system for multimodal model representation optimization, comprising:

[0028] The acquisition module is used to acquire the image data to be segmented;

[0029] The processing module is used to process the acquired images using a pre-trained multimodal model to obtain segmentation results. The training of the multimodal model includes: in the feature extraction stage, feature encoding is performed on the image and text respectively, where visual features are fed into a prior mask generator to generate class-independent prior masks, and text features are used for classification; in the feature interaction stage, visual and text features are enhanced based on content-dependent representation transfer; based on mask-sensitive loss and representation compensation loss, local perceptual capabilities are imparted to the visual encoder through parameter fine-tuning, while replaying the original visual features to avoid overfitting; end-to-end training is used to co-optimize visual-text representations to obtain a better visual-text alignment space; until the model test results meet expectations or the training iterations are reached.

[0030] Thirdly, the present invention provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the open vocabulary segmentation method for multimodal model representation optimization as described in the first aspect.

[0031] Fourthly, the present invention provides a computer device including a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the open vocabulary segmentation method for multimodal model representation optimization as described in the first aspect.

[0032] Fifthly, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the open vocabulary segmentation method for multimodal model representation optimization as described in the first aspect.

[0033] Terminology Explanation:

[0034] Open-vocabulary segmentation is a challenging computer vision task that aims to segment corresponding objects or regions in an image based on arbitrary textual descriptions, even if these object categories do not appear in the training set. This task combines visual feature extraction techniques such as deep convolutional neural networks (CNNs) or visual Transformers with pre-trained multimodal models, achieving arbitrary category processing through cross-modal alignment. It provides advanced image understanding and analysis capabilities in fields such as autonomous driving and healthcare.

[0035] Representation optimization is the process of improving model performance by adjusting and refining how data is represented. Its aim is to capture the essential structure and patterns of data, thereby enhancing the model's learning and reasoning abilities. Key methods include feature engineering (such as normalization, standardization, and dimensionality reduction), parameter fine-tuning, and representation fine-tuning. Representation optimization enables models to understand and process data more accurately, improving predictive performance and generalization ability, and is widely used in various machine learning and data analysis tasks.

[0036] The beneficial effects of this invention are as follows: It better optimizes visual-text representations in multimodal tasks, enabling effective alignment of similar visual-text representation spaces; it proposes a mask-sensitive loss to constrain the classification score and mask quality (IoU score) to remain consistent during parameter fine-tuning, thereby endowing the visual encoder with local perceptual capabilities and greatly improving the performance of pre-trained visual-language models in fine-grained downstream tasks; it introduces the original pre-trained features as representation compensation, reducing overfitting problems on training categories during optimization and ensuring the zero-shot capability of the pre-trained visual-language model during optimization; and it designs a content-dependent representation transfer module to interact with text and visual representations, enabling text representations to be adaptively enhanced for different input images, effectively improving the alignment properties of visual-text in open-vocabulary segmentation.

[0037] The advantages of additional aspects of the invention will be set forth more clearly in the following description or will be learned by practice of the invention. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a diagram of the representation fine-tuning framework for the multimodal pre-trained model described in an embodiment of the present invention.

[0040] Figure 2 This is a diagram of the visual representation compensation structure described in an embodiment of the present invention.

[0041] Figure 3 This is a content-dependent representation transfer structure diagram as described in an embodiment of the present invention.

[0042] Figure 4 This is a flowchart of the open vocabulary segmentation multimodal model training method according to an embodiment of the present invention. Detailed Implementation

[0043] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0044] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0045] It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as here.

[0046] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.

[0047] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0048] To facilitate understanding of the present invention, the present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0049] Those skilled in the art should understand that the accompanying drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily essential for implementing the present invention.

[0050] Open vocabulary segmentation is a challenging multimodal image segmentation task aimed at segmenting images based on arbitrary textual descriptions. Pre-trained visual-language models, such as CLIP, with their well-aligned visual-text embedding spaces, have been increasingly used to solve this task. Traditional open vocabulary segmentation schemes typically freeze the pre-trained visual-language model during training to maintain its zero-shot capability. However, this invention reveals the insensitivity of pre-trained visual-language models to local regions, tending to produce similar predictions for different regions of the same image, thus limiting their application in fine-grained perceptual tasks such as segmentation and detection.

[0051] To address this issue, this invention proposes a visual-text collaborative representation optimization method. We propose a mask-aware loss function to fine-tune the visual representation space, enabling the visual encoder to perceive local regions of the image. Furthermore, a content-dependent transfer module is designed to adaptively enhance the text representation space by interacting with the input image and text. To prevent the pre-trained visual-language model from losing its original zero-shot capability during optimization, a representation compensation strategy is proposed. This strategy replays the feature representations of the original pre-trained model to prevent overfitting on the training data. Therefore, by collaboratively optimizing the visual and text representations of the pre-trained model, the alignment of the visual-text feature spaces is promoted, effectively improving the performance of open-vocabulary segmentation.

[0052] Example 1

[0053] In this embodiment 1, an open vocabulary segmentation system with multimodal model representation optimization is first provided. This system includes: an acquisition module for acquiring image data to be segmented; and a processing module for processing the acquired images using a pre-trained multimodal model to obtain segmentation results. The training of the multimodal model includes: in the feature extraction stage, feature encoding is performed on the image and text respectively, where visual features are fed into a prior mask generator to generate class-independent prior masks, and text features are used for classification; in the feature interaction stage, visual and text features are enhanced based on content-dependent representation transfer; based on mask-sensitive loss and representation compensation loss, local perceptual capabilities are imparted to the visual encoder through parameter fine-tuning, while replaying the original visual features to avoid overfitting; end-to-end training is used to co-optimize visual-text representations to obtain a better visual-text alignment space; until the model test results meet expectations or the training iterations are reached.

[0054] In this embodiment, the above-described system is used to implement an open vocabulary segmentation method with multimodal model representation optimization, including: a practical acquisition module acquires image data to be segmented; a processing module processes the acquired image using a pre-trained multimodal model to obtain segmentation results; wherein, training the multimodal model includes: in the feature extraction stage, feature encoding is performed on the image and text respectively, wherein visual features are fed into a prior mask generator to generate class-independent prior masks, and text features are used for classification; in the feature interaction stage, visual and text features are enhanced based on content-dependent representation transfer; based on mask-sensitive loss and representation compensation loss, local perceptual ability is given to the visual encoder through parameter fine-tuning, while the original visual features are replayed to avoid overfitting; visual-text representations are co-optimized through end-to-end training to obtain a better visual-text alignment space; until the model test results meet expectations or the training iterations are reached.

[0055] In this process, a pre-trained CLIP visual encoder is used to extract features from the input image, and the output of each stage of the visual encoder is represented as F = {F...} i}, i = [0, 1, 2, 3]; MaskFormer is used as the prior mask generator; Hungarian matching is used to optimize the subset of the prior mask that has been matched during training; gradient propagation from the visual encoder to the prior mask generator is terminated to avoid overfitting of the visual encoder on the training categories.

[0056] Introducing mask-aware loss L ma To fine-tune the visual encoder using mask-sensitive features, enabling it to have local perceptual capabilities, including: L ma The goal is to assign high classification scores to high-quality prior masks and low classification scores to low-quality prior masks. Specifically, the intersection-union (IoU) score between the prior mask and the labels is calculated and aligned with the classification scores. Assuming there are k classes in the ground truth, k binary graphs can be generated from the labels, and their IoU score with the prior mask is calculated and denoted as S. IoU The maximum value of the classification score and S IoU There are differences between the maximum values: the maximum value of the classification score usually tends to be close to 1, while the maximum value of S... IoU The maximum value fluctuates between 0.75 and 0.99, and this inconsistency increases the classification score and S. IoU Alignment difficulty between them; therefore, for S IoU The following normalization was introduced:

[0057]

[0058] In addition, L ma Select k categories from the classification scores that exist in the labels, denoted as S.pre and use the SmoothL1 function to combine it with Alignment.

[0059] The visual representation compensation strategy aims to enhance the zero-shot capability of the fine-tuned visual encoder by replaying the original features of the CLIP during the training phase as representation compensation. The original visual features of the CLIP visual encoder, with one parameter frozen, are generated during training as compensation features and incorporated into the fine-tuned visual encoder. The representation compensation loss L... RC To constrain the final layer output of CLIP-V* and the final layer output F of the fine-tuned visual encoder 3 To maintain consistency and enhance the zero-sample capability of the visual encoder.

[0060] Directly and F 3 Pixel-by-pixel alignment leads to the loss of regional differences. Therefore, multi-scale average pooling (AvgPooling) is used to generate multi-scale features, and the consistency of the pooled features is constrained.

[0061] For any feature f, the average pooling operation with a grid size of k×k is expressed as:

[0062]

[0063] exist and F 3 Based on this, multi-scale average pooling with a grid size of K = {1, 2, 4} is used to transform it into a grid of {1×1, 2×2, 4×4}, denoted as and F p ,Right now: F p =AvgPooling(F 3 ,K); where L RC SmoothL1 was used to minimize and F p Differences:

[0064] The content-dependent representation transfer module includes a series of Transformer layers that undergo cross-attention with visual features; among them, the final layer features F of the visual encoder are transferred... 3 And the text representation T is used as input for content-dependent representation transfer. First, F 3 The reordering operation is performed in the spatial dimension, followed by processing T and F using n consecutive Transformer layers. 3 Simultaneously, residual connections are introduced, and this process is expressed as:

[0065] Ti+1 =TransLayer i (T i ,F 3 )+T i i = 1, 2, ... L

[0066] The output of content-dependent representation transfer is the optimized text representation, denoted as . The Transformer layer is represented by the following formula:

[0067]

[0068] Where Que(·), Key(·), and Val(·) represent linear projections, and d is the dimension of the input vector.

[0069] Example 2

[0070] Existing pre-trained large-scale visual-text models lack local perception capabilities, making them difficult to directly apply to segmentation tasks. Therefore, this embodiment designs a mask-sensitive loss to fine-tune the visual-text pre-trained model, enabling it to possess local perception capabilities and enhancing the application value of the pre-trained large-scale model in segmentation tasks. A paradigm for joint visual-text optimization is designed, enabling effective alignment of the visual-text space. This joint optimization breaks through the paradigm of the pre-trained visual-text representation space to some extent. Therefore, the optimized visual-text representation can better serve open-vocabulary segmentation tasks and meet the requirements of real-world scenarios.

[0071] This embodiment proposes a novel open vocabulary segmentation algorithm—a representation fine-tuning method based on a multimodal pre-trained model, the overall structure of which is as follows: Figure 1 As shown, this method uses the multimodal pre-trained model CLIP for visual and text encoding, and trains a prior mask generator (Proposal Generator) to generate class-independent prior masks. Simultaneously, a mask-aware loss (L...) is designed. ma This paper proposes a collaborative optimization method for the visual and textual representations of CLIP, enhancing the local perceptual capabilities of the visual encoder. Furthermore, a Content-Dependent Transfer module is designed to adaptively enhance the text representation space by interacting with the input image and text. To maintain CLIP's zero-shot capability while optimizing representations, a Representation Compensation strategy is proposed. This strategy replays the original CLIP features, preventing overfitting of the model on the training set during training.

[0072] In this embodiment, a visual encoder and a priori mask generator (Proposal Generator) are proposed. A pre-trained CLIP visual encoder is used to extract features from the input image. The output of each stage of the visual encoder is represented as F = {F...} i}, i = [0, 1, 2, 3]. Where F 0 ,F 1 ,F 2 ,F 3 The downsampling ratios relative to the input image are {4, 8, 16, 32}. MaskFormer is used as the prior mask generator. During training, Hungarian matching is used to optimize the subsets of completed matches in the prior mask. Therefore, the trained prior mask generator has strong generalization ability and can segment targets of unknown categories. Furthermore, gradient propagation from the visual encoder to the prior mask generator is terminated during training, thus avoiding overfitting of the visual encoder to the training categories.

[0073] In this embodiment, a mask-aware loss (L) is proposed. ma Because visual encoders using CLIP pre-trained parameters are insensitive to local regions of an image—that is, when the prior mask contains more background regions than foreground objects—CLIP may tend to classify it as belonging to the foreground category. To overcome this limitation of CLIP, this embodiment introduces a mask-aware loss (L... ma This is used to fine-tune the mask-sensitive features of the visual encoder, giving it local perception capabilities. ma The goal is to assign high classification scores to high-quality prior masks and low classification scores to low-quality prior masks. Specifically, the intersection-over-union (IoU) score between the prior mask and the annotations is calculated and aligned with the classification score to enable CLIP to become mask-aware. Assuming there are k classes in the ground truth, k binary graphs can be generated from the annotations, and their IoU scores with the prior mask (denoted as S) are calculated. IoU Among them, the maximum value of the classification score and S IoU There are differences between the maximum values: the maximum value of the classification score usually tends to be close to 1, while the maximum value of S... IoU The maximum value fluctuates between 0.75 and 0.99. This inconsistency increases the classification score and S. IoU The difficulty of alignment between them. Therefore, for S IoU The following normalization technique was introduced:

[0074]

[0075] In addition, L ma Select the k categories that exist in the classification scores (denoted as S).pre ), and use the SmoothL1 function to combine it with Alignment. Therefore, L ma This can be expressed as follows:

[0076]

[0077] SmoothL1 can be expressed as follows:

[0078]

[0079] The visual representation compensation (RC) strategy aims to enhance the zero-shot capability of fine-tuning the visual encoder by replaying the original features of CLIP during the training phase as representation compensation. Its detailed structure is shown in the diagram below. Figure 2 As shown, a CLIP visual encoder with one parameter frozen (denoted as CLIP-V*) generates the original visual features as compensation features during training and incorporates them into the fine-tuned visual encoder. Specifically, a representation compensation loss (L...) is designed... RC To constrain the CLIP-V* final layer output and the final layer output of the fine-tuned visual encoder (F 3 To maintain consistency and enhance the zero-sample capability of the visual encoder. However, directly using and F 3 Pixel-by-pixel alignment leads to the loss of regional diversity (the optimized visual encoder loses its region perception capability). Therefore, this embodiment uses multi-scale average pooling (AvgPooling) to generate multi-scale features and constrains the consistency of the pooled features.

[0080] For any feature f, the average pooling operation with a grid size of k×k can be expressed as:

[0081]

[0082] In this embodiment, and F 3 Based on this, multi-scale average pooling with a grid size of K = {1, 2, 4} is used to transform it into a grid of {1×1, 2×2, 4×4}, denoted as and F p .Right now: F p =AvgPooling(F 3 ,K). L RC SmoothL1 was used to minimize and F p Differences:

[0083]

[0084] This embodiment proposes Content-Dependent Transfer (CDT). Given a set of category nouns C = {C1, C2, C3, ..., C...} n We first use predefined text templates to generate sentences corresponding to these category nouns, for example: "a C i The photo; there is a C in the scene. i ...”. Next, these sentences are input into a text encoder to obtain the corresponding text vector for each sentence. Finally, the text vectors of the same category are averaged to obtain the text representation (denoted as T) for each category.

[0085] To optimize CLIP text representation T, this embodiment proposes a content-dependent representation transfer (CDT) module, which includes a series of Transformer layers and visual features F. 3 Perform cross-attention. Details of content-dependent representation transfer include... Figure 3 As shown. In this embodiment, the final layer features F of the visual encoder are... 3 The text representation T is used as input for content-dependent representation transfer. First, F... 3 A flattening operation is performed in the spatial dimension. Then, n consecutive Transformer layers are used to process T and F. 3 Simultaneously, residual connections are introduced. This process can be represented as:

[0086] T i+1 =TransLayer i (T i ,F 3 )+T i ,i=1,2,...L (7)

[0087] In this embodiment, L is set to 2. The output of content-dependent representation transfer is the optimized text representation (denoted as...). The Transformer layer can be represented by the following formula:

[0088]

[0089] Where Que(·), Key(·), and Val(·) represent linear projections, and d is the dimension of the input vector. It should be noted that during training, this embodiment freezes the text encoder; only the Transformer layer is trained to optimize the text representation. Therefore, this embodiment establishes a parameter-efficient text representation optimization strategy, resulting in an optimized text representation. It depends on the input image.

[0090] like Figure 4 As shown, the training method for the model described in this embodiment includes the following steps:

[0091] S1: In open-vocabulary segmentation, the input is an RGB three-channel image of any category. Given two category sets C train and C test C train and C test The number of categories is not equal (C train ≠C test The model in C train Training is performed on it, and directly on C. test Tested on C. Typically, C... train and C test Described by category nouns (e.g., sky, ocean, mountains, etc.);

[0092] S2: In the data processing stage, the input image is scaled to 1024×1024 after data augmentation processes such as flipping and random cropping. Previous work has shown that higher resolution inputs in open-vocabulary segmentation help capture local image properties;

[0093] S3: In the feature extraction stage, in this embodiment, CLIP is used to encode visual and text features respectively. The visual features are fed into a priori mask generator to generate class-independent prior masks, and the text features are used for classification.

[0094] S4: In the feature interaction stage, this embodiment designs content-dependent representation transfer to enhance visual text features;

[0095] S5: Design mask-sensitive loss and representation compensation loss, and give the visual encoder local perception ability through parameter fine-tuning, while replaying the original visual features to avoid overfitting;

[0096] S6: Co-optimize visual-text representations through end-to-end training to obtain a better visual-text alignment space;

[0097] S7: Repeat S1-S6 until the model test results meet expectations or the number of training iterations is reached.

[0098] Example 3

[0099] This embodiment 3 provides a non-transitory computer-readable storage medium for storing computer instructions. When executed by a processor, the computer instructions implement the open vocabulary segmentation method for multimodal model representation optimization as described above. The method includes:

[0100] Obtain the image data to be segmented;

[0101] The acquired images are processed using a pre-trained multimodal model to obtain segmentation results. The training of the multimodal model includes: in the feature extraction stage, feature encoding is performed on the image and text respectively, where visual features are fed into a prior mask generator to generate class-independent prior masks, and text features are used for classification; in the feature interaction stage, visual and text features are enhanced based on content-dependent representation transfer; based on mask-sensitive loss and representation compensation loss, local perceptual capabilities are imparted to the visual encoder through parameter fine-tuning, while replaying the original visual features to avoid overfitting; end-to-end training is used to co-optimize visual-text representations to obtain a better visual-text alignment space; until the model test results meet expectations or the training iterations are reached.

[0102] Example 4

[0103] This embodiment 4 provides a computer device, including a memory and a processor, wherein the processor and the memory communicate with each other, and the memory stores program instructions executable by the processor. The processor calls the program instructions to execute the open vocabulary segmentation method for multimodal model representation optimization as described above, the method including:

[0104] Obtain the image data to be segmented;

[0105] The acquired images are processed using a pre-trained multimodal model to obtain segmentation results. The training of the multimodal model includes: in the feature extraction stage, feature encoding is performed on the image and text respectively, where visual features are fed into a prior mask generator to generate class-independent prior masks, and text features are used for classification; in the feature interaction stage, visual and text features are enhanced based on content-dependent representation transfer; based on mask-sensitive loss and representation compensation loss, local perceptual capabilities are imparted to the visual encoder through parameter fine-tuning, while replaying the original visual features to avoid overfitting; end-to-end training is used to co-optimize visual-text representations to obtain a better visual-text alignment space; until the model test results meet expectations or the training iterations are reached.

[0106] Example 5

[0107] This embodiment 5 provides an electronic device, including: a processor, a memory, and a computer program; wherein, the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the open vocabulary segmentation method for multimodal model representation optimization as described above, the method including:

[0108] Obtain the image data to be segmented;

[0109] The acquired images are processed using a pre-trained multimodal model to obtain segmentation results. The training of the multimodal model includes: in the feature extraction stage, feature encoding is performed on the image and text respectively, where visual features are fed into a prior mask generator to generate class-independent prior masks, and text features are used for classification; in the feature interaction stage, visual and text features are enhanced based on content-dependent representation transfer; based on mask-sensitive loss and representation compensation loss, local perceptual capabilities are imparted to the visual encoder through parameter fine-tuning, while replaying the original visual features to avoid overfitting; end-to-end training is used to co-optimize visual-text representations to obtain a better visual-text alignment space; until the model test results meet expectations or the training iterations are reached.

[0110] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0111] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0112] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0113] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0114] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solutions disclosed in the present invention, various modifications or variations that can be made by those skilled in the art without creative effort should be included within the scope of protection of the present invention.

Claims

1. An open vocabulary segmentation method for multimodal model representation optimization, characterized in that, include: Obtain the image data to be segmented; The acquired images are processed using a pre-trained multimodal model to obtain segmentation results. Training the multimodal model includes: in the feature extraction stage, feature encoding is performed on the image and text respectively, where visual features are fed into a prior mask generator to generate class-independent prior masks, and text features are used for classification; in the feature interaction stage, visual and text features are enhanced based on content-dependent representation transfer; based on mask-sensitive loss and representation compensation loss, local perceptual capabilities are imparted to the visual encoder through parameter fine-tuning, while replaying the original visual features to avoid overfitting; end-to-end training is used to co-optimize visual-text representations to obtain a better visual-text alignment space; training continues until the model test results meet expectations or the training iterations are reached; content-dependent representation transfer includes a series of Transformer layers with cross-attention to visual features; wherein, the final layer features of the visual encoder are... and text representation As input for content-dependent representation transfer, firstly... Perform a reordering operation in the spatial dimension, and then use... Processing a series of Transformer layers and Simultaneously, residual connections are introduced, and the process is represented as follows: , i = 1, 2, ... L; The output of content-dependent representation transfer is the optimized text representation, denoted as The Transformer layer is represented by the following formula: r(a, b) = SoftMax( ) ; in , and This represents a linear projection, where d is the dimension of the input vector; In addition, mask-aware loss was introduced. To fine-tune the visual encoder using mask-sensitive features, enabling it to possess local perceptual capabilities, including: The goal is to assign high classification scores to high-quality prior masks and low classification scores to low-quality prior masks. Specifically, the intersection-union ratio (IU) between the prior mask and the label is calculated and aligned with the classification score. Assume that the true values ​​include... If there are several categories, they can be generated from the annotations. A binary graph is generated, and its IoU score with the prior mask is calculated and denoted as [the score]. ; Among them, mask-perceived loss The statement is as follows: ； Maximum and maximum category scores There are differences between the maximum values: the maximum value of the classification score usually tends to be close to 1, while... The maximum value fluctuates between 0.75 and 0.99, and this inconsistency increases the classification score and The difficulty of alignment between them; therefore, for The following normalization was introduced: ； also, Select from category scores The categories that exist in each label are denoted as and use The function combines it with Alignment; in The following statement is made: 。 2. The open vocabulary segmentation method for multimodal model representation optimization according to claim 1, characterized in that, Features are extracted from the input image using a pre-trained CLIP visual encoder, and the output of each stage of the visual encoder is represented as follows: Use MaskFormer as the prior mask generator; During training, Hungarian matching is used to optimize the subset of completed matches in the prior mask; gradient propagation from the visual encoder to the prior mask generator is terminated, thus avoiding overfitting of the visual encoder on the training categories.

3. The open vocabulary segmentation method for multimodal model representation optimization according to claim 1, characterized in that, The visual representation compensation strategy aims to enhance the zero-shot capability of the fine-tuned visual encoder by replaying the original features of the CLIP during the training phase as representation compensation. The original visual features of the CLIP visual encoder, with one parameter frozen, are generated during training as compensation features and incorporated into the fine-tuned visual encoder. The representation compensation loss... To constrain the final layer output of CLIP-V* and the final layer output of the fine-tuned visual encoder To maintain consistency and enhance the zero-sample capability of the visual encoder.

4. The open vocabulary segmentation method for multimodal model representation optimization according to claim 3, characterized in that, directly and Pixel-by-pixel alignment leads to the loss of regional differences. Therefore, multi-scale average pooling (AvgPooling) is used to generate multi-scale features, and the consistency of the pooled features is constrained. For any feature f, the grid size is The average pooling operation is represented as: (f, k) , exist and Based on the grid size of Multi-scale average pooling is transformed into The grid, denoted as and ,Right now: ( , K), ( , K); where, SmoothL1 was used to minimize and Differences: 。 5. A system based on the open vocabulary segmentation method for multimodal model representation optimization as described in any one of claims 1-4, characterized in that, include: The acquisition module is used to acquire the image data to be segmented; The processing module is used to process the acquired images using a pre-trained multimodal model to obtain segmentation results. The training of the multimodal model includes: in the feature extraction stage, feature encoding is performed on the image and text respectively, where visual features are fed into a prior mask generator to generate class-independent prior masks, and text features are used for classification; in the feature interaction stage, visual and text features are enhanced based on content-dependent representation transfer; based on mask-sensitive loss and representation compensation loss, local perceptual capabilities are imparted to the visual encoder through parameter fine-tuning, while replaying the original visual features to avoid overfitting; end-to-end training is used to co-optimize visual-text representations to obtain a better visual-text alignment space; until the model test results meet expectations or the training iterations are reached.

6. A computer device, characterized in that, The method includes a memory and a processor, the processor and the memory communicating with each other, the memory storing program instructions executable by the processor, and the processor calling the program instructions to execute the open vocabulary segmentation method for multimodal model representation optimization as described in any one of claims 1-4.

7. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the open vocabulary segmentation method for multimodal model representation optimization as described in any one of claims 1-4.

Images