A self-supervised image representation extraction model enhancement method based on visual fundamental models
By learning semantic associations through channel mask modeling, and optimizing the image representation extraction model, the problem of high resource consumption in existing technologies is solved, and efficient visual feature reconstruction and performance improvement are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2023-11-20
- Publication Date
- 2026-05-26
AI Technical Summary
Existing visual self-supervised methods suffer from high resource consumption and time overhead when training larger visual models, and lack effective utilization of the trained model, making it difficult to effectively remove semantic information.
The channel masking modeling method is adopted. By learning semantic associations on feature channels, the shared layer regularization module of the base model and the extended model is used to mask and reconstruct visual features, thereby optimizing the parameters of the image representation extraction model.
While reducing computational and time costs, it improves the model's performance on downstream vision tasks, maintains the capabilities of the base model, and provides high-quality training data.
Smart Images
Figure CN117456307B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image model construction technology, and in particular relates to a method for enhancing a self-supervised image representation extraction model of a visual basic model. Background Technology
[0002] Self-supervised learning trains models using a large amount of unlabeled visual data by constructing different proxy tasks. Through learning from these proxy tasks, the visual model can learn good data representations, thus achieving better performance for downstream visual tasks. Currently, the most mainstream visual self-supervised methods include contrastive learning and image masking modeling. Contrastive learning constructs different perspectives for a natural image using different data augmentation methods, treating different perspectives of the same image as positive sample pairs and data from different images as negative sample pairs. Contrastive learning learns data representations by increasing the similarity of positive sample pairs and decreasing the similarity of negative sample pairs in the representation space. Recently, image masking modeling pre-training methods have received widespread attention due to their excellent transfer performance and scalability for downstream tasks. These methods divide natural images into disjoint image patches, randomly remove some image patches, and the model uses information from the remaining image patches to predict the signals of the removed images. In 2020, Chen et al. proposed the Image Generation Pre-training (iGPT) model, which predicts masked low-frequency image pixel information through autoregression. In 2021, Bao et al. proposed the Bidirectional Encoder Representation (BEiT) model for image Transformers. They utilized an additional word segmenter to convert image data into visual feature vectors, then allowed the visual model to predict the visual feature vectors of missing image patches. In the same year, He et al. proposed the Masked Autoencoder (MAE) model, and Xie et al. proposed the Simple Image Masking Modeling (SimMIM) model. MAE designed an asymmetric encoder-decoder structure, allowing the encoder to learn data representations by removing a large proportion of random image patches and reconstructing them using the decoder. SimMIM employed a simple strategy, adding a linear layer after the encoder to predict missing image pixels. Although current image masking modeling methods exhibit good generalization and scalability, they lack the ability to utilize pre-trained models. When using these methods to train a larger visual model, they often result in significant resource consumption and time overhead.
[0003] Patent document CN116704221 A discloses an image processing method, apparatus, device, and computer-readable storage medium, comprising: acquiring a reference image and acquiring a similar image to the reference image; determining difference information between the reference image and the similar image; determining a target mask image in the reference image for the difference information; expanding the target mask image based on the difference information to obtain difference description text; and performing local adjustments on the similar image based on the target mask image, the difference description text, and the reference image to obtain an adjusted target image.
[0004] Patent document CN 116664409 A discloses an image super-resolution reconstruction method, apparatus, computer device, and storage medium. First, an initial resolution image is input into the linear activation module of a super-resolution reconstruction model to obtain shallow image features of the initial resolution image. Second, a first lightweight feature extraction module extracts features from the shallow image features to obtain first deep image features. Then, a fine-grained sparse mask branch sets a mask for each element in the first deep image features to obtain the corresponding fine-grained sparse mask. Next, a ghost feature extraction branch extracts ghost features based on the fine-grained sparse mask and the first deep image features to obtain the corresponding second deep image features. Finally, super-resolution reconstruction is performed based on the initial resolution image, shallow image features, and second deep image features to obtain the target resolution image.
[0005] The aforementioned patents all employ the removal of image information in the spatial dimension to construct an image completion task. For visual features, local visual features in space often have higher similarity to the original image. Simply removing spatial features cannot effectively remove semantic information. Summary of the Invention
[0006] The purpose of this invention is to provide a self-supervised image representation extraction model enhancement method for visual basic models. This method can build a high-quality image training set in a short time, thereby providing more comprehensive performance guidance for subsequent models to complete downstream tasks.
[0007] To achieve the objectives of this invention, a method for enhancing a self-supervised image representation extraction model based on a visual fundamental model is provided, comprising the following steps:
[0008] Image information is acquired and input into the feature extractor of the image representation extraction model to be trained, so as to obtain the corresponding visual features;
[0009] The visual features are masked along the channel dimension using a channel mask vector to obtain the corresponding incomplete visual features.
[0010] The incomplete visual features are input into a pre-built extended model, which includes a feature extraction module, a self-attention module, and a prediction module. The feature extraction module is used to extract semantic features of the input incomplete visual features. The self-attention module is used to perform attention operations on the semantic features and the incomplete visual features. The prediction module generates semantic features of the missing part based on the attention operation results and repairs the incomplete visual features based on the generated semantic features to output the corresponding reconstructed visual features.
[0011] The visual features and reconstructed visual features of image information are constrained in the latent representation space to update the network parameters of the image representation extraction model to be trained until the iteration termination condition is met to obtain the enhanced image representation extraction model.
[0012] This invention uses visual features as input and output, and learns the semantic associations distributed on the feature channels through the proposed feature channel masking modeling method. This not only completely masks the image information, but also reduces the difficulty of the pre-training task, effectively removes the semantic information in the visual features, and provides high-quality training data for subsequent model training.
[0013] Specifically, the generation process of the visual features is as follows:
[0014] The given image information is divided into multiple non-overlapping image blocks along the channel dimension, and the image blocks are input into the feature extractor to output the corresponding image block vector;
[0015] The category representation vector corresponding to the image patch is embedded into the header of the corresponding image patch vector, and the position of each vector is encoded to construct the complete image patch vector.
[0016] The constructed image block vectors are subjected to attention operations to obtain the corresponding visual features.
[0017] Specifically, the positional expression of the channel mask vector is as follows:
[0018]
[0019] Among them, z i It is the feature map of the i-th channel of the visual feature. The position of the channel mask vector is represented by δ(·), S represents the set of positions of the removed feature channels, and δ(·) represents the indicator function, which is 1 when i∈S and 0 otherwise.
[0020] Specifically, the latent representation space is constructed in a layer regularization module shared between the extended model and the image representation extraction model to be trained. The layer regularization module is used to prevent the extended model and the image representation extraction model to be trained from overfitting during the training process.
[0021] Specifically, the parameters of the image representation extraction model to be trained are updated by calculating the channel feature map loss between visual features and reconstructed visual features.
[0022] Specifically, the expression for the channel feature map loss is as follows:
[0023]
[0024] Where σ(S) represents the number of feature channels removed, LN represents the shared layer regularization module, and LN(z) i LN(z) represents the image features of the i-th channel after regularization of the original image features. p ) i This represents the image feature of the i-th channel after regularization operations on the predicted image features.
[0025] Specifically, when updating the image representation extraction model to be trained, the AdamW algorithm is used to optimize the parameters in the extended model.
[0026] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0027] By masking the output of the backbone network of an existing model, semantic associations of visual features can be learned within the feature channels. This approach is applicable to any existing pre-trained model and can effectively train the parameters of its extended model. While reducing computational and time costs, it can effectively improve the performance of the base model on downstream visual tasks, and the extended model can inherit the capabilities of the base model. Attached Figure Description
[0028] Figure 1 A schematic diagram of the image representation extraction model enhancement method for the visual basic model provided in this embodiment;
[0029] Figure 2 A flowchart illustrating the processing of visual features provided in this embodiment;
[0030] Figure 3 This is a schematic diagram of the channel mask vector provided in this embodiment. Detailed Implementation
[0031] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.
[0032] In some processes described in the specification, claims, and the foregoing drawings, multiple steps are included in a specific order. However, it should be clearly understood that these steps may be performed in any order or in parallel. The step numbers are merely used to distinguish different steps and do not represent any execution order. Furthermore, descriptions such as "first" and "second" in this document are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0034] like Figure 1 As shown, this embodiment presents a self-supervised image representation extraction model enhancement method based on a visual base model. This method utilizes a base model, an extended model, a channel mask vector, and a shared layer regularization module to construct a visual feature mask modeling framework. This framework takes high-dimensional visual features as input and output and learns semantic associations distributed across feature channels through the proposed feature channel mask modeling method. The base model adopts a visual Transformer model structure, which consists of a linear mapping module, a category representation vector, positional encoding, and a series of self-attention computation modules. The base model is pre-learned through contrastive learning, language-image contrastive pre-training, or image mask modeling; all its parameters remain fixed during training. The extended model also contains a series of self-attention computation modules, whose parameters are learned during training to capture semantic associations between visual features. Unlike image mask modeling, the channel mask vector is a learnable vector with spatial dimensional structure; different lost feature channel positions share the same channel mask vector. To improve training stability, a shared layer regularization module is applied to both image features and predicted image features before calculating the loss.
[0035] The specific operation is as follows:
[0036] like Figure 2 As shown, the visual features of an image are extracted using a base model trained on multiple iterations: First, a given image x is divided into N disjoint image patches. Then, the base model uses a linear mapping to map these image patches into a series of image patch vectors. The number of channels in each image patch vector is C. Then, the category representation vector... The vectors are inserted at the beginning of these image patch vectors, and a positional encoding is added to each vector. The self-attention computation module of the base model takes the processed result as input and outputs the visual features z of the image.
[0037] Masking visual semantic features. Remove 50% of the channel feature map along the channel dimension from the visual feature z, and then replace the feature at the missing channel position with the channel mask vector m.
[0038] like Figure 3 The diagram shown is a schematic of the channel mask vector.
[0039] Incomplete visual features z c Channel features at the i-th position The calculation method is as follows:
[0040]
[0041] Where z i Let be the feature map of the i-th channel of the visual feature, S represent the set of locations of the removed feature channels, and δ(·) represent the indicator function, where δ = 1 when i ∈ S, and δ = 0 otherwise.
[0042] The extended model reconstructs the incomplete visual features. The incomplete visual features z... c As input to the extended model, after a series of self-attention computation modules, the extended model learns the associations between semantics, predicts the lost channel feature maps using the remaining semantic information, and outputs the reconstructed visual features z. p .
[0043] Calculate the loss and optimize the model. Use a shared layer regularization module to constrain the visual features of the image and the reconstructed visual features in the latent representation space, and then calculate the loss for the lost channel feature maps:
[0044]
[0045] σ(S) represents the number of feature channels removed, and LN represents the shared layer regularization module. The loss is calculated, and the AdamW algorithm is used to optimize the extended model. This process is repeated until the model's loss converges.
[0046] In addition, to illustrate the gain effect of the method provided by the present invention, a performance comparison with conventional models is also provided.
[0047] This paper compares our method's model (MFM) with the commonly used CLIP model. The CLIP model, short for Contrastive Language-Image Pre-Training, is a pre-trained neural network model used for matching images and text. This model typically uses image masks for data-driven training.
[0048] Table 1 shows a comparison of the zero-shot classification performance of the two models.
[0049] Table 1
[0050] method IN-1K IN-Adv. IN-Ren. IN-Ske. IN-V2 ObjectNet CIFAR-10 CIFAR-100 Avg. CLIP 75.5 70.7 87.9 59.6 69.8 69.0 95.7 76.5 75.6 MFM 75.8 70.8 88.2 60.1 70.0 69.9 96.1 77.9 76.1
[0051] As can be seen from the performance comparisons across different test sets, the model provided by this invention shows a certain degree of performance improvement.
[0052] Meanwhile, the robustness and generalization of the two in the test are compared in Table 2.
[0053] Table 2
[0054] method IN-1K IN-Adv. IN-Ren. IN-Ske. IN-V2 Avg. CLIP 84.9 45.8 63.1 48.8 74.9 63.5 MFM 85.8 53.2 67.0 51.8 75.7 66.7
[0055] In addition, a performance comparison of the two models after fine-tuning for downstream tasks is provided, and the results are shown in Table 3.
[0056] Table 3
[0057]
[0058] The data in the table clearly demonstrate that the performance of the model enhanced by this invention is indeed superior to that of the traditional CLIP model.
[0059] The above provides a detailed description of an image representation extraction model enhancement method provided by the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for enhancing a visual fundamental model through self-supervised image representation extraction, characterized in that, Includes the following steps: Image information is acquired and input into the feature extractor of the image representation extraction model to be trained, so as to obtain the corresponding visual features; The visual features are masked along the channel dimension using a channel mask vector to obtain the corresponding incomplete visual features. The incomplete visual features are input into a pre-built extended model, which includes a feature extraction module, a self-attention module, and a prediction module. The feature extraction module is used to extract semantic features of the input incomplete visual features. The self-attention module is used to perform attention operations on the semantic features and the incomplete visual features. The prediction module generates semantic features of the missing part based on the attention operation results and repairs the incomplete visual features based on the generated semantic features to output the corresponding reconstructed visual features. In the latent representation space, the visual features and reconstructed visual features of image information are constrained to update the network parameters of the image representation extraction model to be trained until the iteration termination condition is met to obtain the enhanced image representation extraction model. The latent representation space is constructed in a shared layer regularization module between the extended model and the image representation extraction model to be trained. The layer regularization module is used to prevent the extended model and the image representation extraction model to be trained from overfitting during the training process. The parameters of the image representation extraction model to be trained are updated by calculating the channel feature map loss between visual features and reconstructed visual features.
2. The image representation extraction model enhancement method for visual fundamental models according to claim 1, characterized in that, The process of generating the visual features is as follows: The given image information is divided into multiple non-overlapping image blocks along the channel dimension, and the image blocks are input into the feature extractor to output the corresponding image block vector; The category representation vector corresponding to the image patch is embedded into the header of the corresponding image patch vector, and the position of each vector is encoded to construct the complete image patch vector. The constructed image block vectors are subjected to attention operations to obtain the corresponding visual features.
3. The image representation extraction model enhancement method for visual fundamental models according to claim 1, characterized in that, The ratio of the channel mask vector to the visual feature vector is 1:
2.
4. The image representation extraction model enhancement method for a self-supervised visual fundamental model according to claim 1, characterized in that, The positional expression for the channel mask vector is as follows: ;in, It is the first visual feature Feature map of each channel Indicates the position of the channel mask vector. This represents the set of locations of the feature channels that were removed. Indicates an indicator function, when , ,otherwise .
5. The method for enhancing the image representation extraction model of the visual fundamental model according to claim 1, characterized in that, The expression for the channel feature map loss is as follows: ;in, This indicates the number of feature channels that were removed. This indicates a shared layer regularization module. This indicates the result of regularization operation on the features of the original image. Image features of each channel This indicates the result of regularization operation on the predicted image features. Image features of each channel.
6. The image representation extraction model enhancement method for a self-supervised visual fundamental model according to claim 1, characterized in that, When updating the image representation extraction model to be trained, the AdamW algorithm is used to optimize the parameters in the extended model.