Self-supervised training method of cross-distillation-based vit encoder

Abstract

The application provides a self-supervised training method of a cross-distillation-based ViT encoder, comprising generating a teacher and student view for image sampling and composing a view pair, sampling a set of occlusion positions for the student view; the teacher view is extracted by a teacher ViT encoder and a projection head to obtain a teacher [CLS] embedding and an image block embedding sequence, and is mapped to a teacher [CLS] target distribution and an image block target distribution; the student view is extracted by a student ViT encoder and a projection head after being occluded to obtain a student [CLS] embedding and an image block embedding sequence, and is mapped to a student [CLS] prediction distribution and an image block prediction distribution; for the occlusion positions, the teacher image block target distribution and the teacher [CLS] target distribution of the corresponding positions are convexly combined into a mixed teacher target; an image-level distillation loss is determined according to the student [CLS] prediction distribution and the teacher [CLS] target distribution, a cross-distillation loss is determined according to the student image block prediction distribution of the occlusion positions and the mixed teacher target, and a total loss is obtained in combination to update the student network, and the teacher network is updated by an exponential moving average.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, specifically to the field of self-supervised learning for image classification and image segmentation, and more specifically, to a self-supervised training method for a ViT encoder based on cross-distillation. Background Technology

[0002] Self-supervised learning pre-trains visual models on unlabeled images, enabling them to learn general visual representations without relying on manual category labels. In recent years, teacher-student self-distillation methods based on Vision Transformers (ViTs) have demonstrated outstanding performance in image-level tasks such as image classification and have gradually become an important technical approach for training fundamental visual models. These models typically include class tokens (image-level tokens, hereinafter referred to as [CLS] tokens) and patch tokens (hereinafter referred to as patch tokens). The former primarily aggregates image-level semantics, while the latter is directly used for intensive prediction tasks such as semantic segmentation, depth estimation, and video object segmentation.

[0003] Existing self-distillation methods typically set both image-level [CLS] distillation loss and masked image patch distillation loss simultaneously. However, image-level semantics are not uniformly transferred to all image patch labels. The inventors found that in self-supervised ViT, [CLS] labels tend to converge global semantics from a small number of discriminative regions, which can be called representative image patches; while low-attention, non-representative image patches are equally important for dense prediction, they are more prone to semantic deficiencies and prediction errors.

[0004] Therefore, the existing technology has at least the following shortcomings: First, the image-level self-distillation signal is mainly transmitted through [CLS] attention readout, and the global semantic gradient obtained by low-attention image patches is weak; Second, ordinary occluded image patch distillation mainly learns local image patch targets and lacks direct global semantic correction; Third, if this problem is solved by changing the backbone structure or introducing manual annotation, the training cost will be increased and the universality of self-supervised training will be weakened.

[0005] It should be noted that the background information presented here is only for illustrating relevant information about the present invention to aid in understanding the technical solution of the present invention, and does not imply that the relevant information is necessarily prior art. The relevant information was submitted and disclosed together with the present invention, and should not be considered prior art unless there is evidence that the relevant information was disclosed before the filing date of the present invention. Summary of the Invention

[0006] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a self-supervised training method for ViT encoders based on cross-distillation.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] According to a first aspect of the present invention, a self-supervised training method for a ViT encoder is provided, comprising the steps of: S1, performing augmented sampling on all training images in the current batch to generate teacher views and student views and forming view pairs, and sampling the location set of occluded image patches for at least a portion of the student views, wherein the two views in each view pair correspond to the same spatial region of the same training image and use the same geometric transformation during augmented sampling; S2, inputting the teacher views into the teacher network, extracting the teacher [CLS] embedding and teacher image patch embedding sequence from the input view (teacher view) through its ViT encoder, and mapping the teacher [CLS] embedding to the teacher [CLS] target distribution through its projection head, and mapping the teacher image patch embedding sequence to the target distribution of each teacher image patch; S3, occluding the corresponding image patches of the student views according to their location sets and inputting them into the student network, extracting the student [CLS] embedding and student image patch embedding sequence from the input view (student view) through its ViT encoder. S4. For each occluded position in the location set, the teacher image block target distribution and the teacher [CLS] target distribution at the corresponding position are convexly combined according to a preset ratio to obtain the mixed teacher target at the occluded position; S5. The image-level distillation loss is determined according to the student [CLS] target distribution and the teacher [CLS] target distribution at the occluded position in the view pair, and the cross-distillation loss is determined according to the student image block target distribution at the occluded position and the mixed teacher target at the corresponding occluded position in the view pair, and the total loss is obtained according to the image-level distillation loss and the cross-distillation loss; S6. The ViT encoder and projector head parameters of the student network are updated according to the total loss, and the ViT encoder and projector head parameters of the teacher network are updated by exponential moving average. This scheme can achieve at least the following beneficial technical effects: by directly injecting the teacher's [CLS] global semantic distribution into occluded image patches, low-attention, non-representative image patches can obtain semantic correction that is not explicitly related to attention quality, thereby improving the spatial integrity of dense representations at the image patch level; without changing the standard self-distillation framework or using manual labels, it can improve the quality of dense representations extracted by the self-supervised ViT encoder; after training, the ViT encoder has improved transfer performance on various downstream tasks and maintains model robustness when representative image patches are occluded.

[0009] Optionally, the hybrid teacher target is determined as follows: A first weighting coefficient is obtained for the teacher [CLS] target distribution, and a second weighting coefficient is obtained for the teacher image patch target distribution, wherein the sum of the first and second weighting coefficients is 1 and both are positive values; the first weighting coefficient is multiplied by the teacher [CLS] target distribution to obtain a global target component; the second weighting coefficient is multiplied by the teacher image patch target distribution at the corresponding occluded position in the teacher image patch target distribution to obtain a local target component; based on the global target component and the local target component, the hybrid teacher target at the occluded position is obtained. This scheme can achieve at least the following beneficial technical effects: by explicitly setting the first and second weighting coefficients, the proportion of global semantics and local semantics in the hybrid teacher target can be precisely controlled; the construction process of the hybrid target is interpretable and adjustable, ensuring that the hybrid teacher target is a convex combination of probability distributions, thus improving encoder performance.

[0010] Optionally, blended teacher objectives can be determined as follows:

[0011]

[0012] in, For the first The mixed teacher objectives corresponding to each occlusion location The first weighting coefficient, The value range of is (0,1). The difference obtained is the second weighting coefficient. For Teacher View The determined first The distribution of teacher image patch targets at each occlusion location. For the teacher view The teacher [CLS] target distribution, The occlusion location index is given in the location set. This scheme can achieve at least the following beneficial technical effects: it enables rapid fusion of the teacher image patch target distribution and the teacher [CLS] target distribution through a simple linear weighting formula, reducing computational overhead; within a specified value range, each occluded image patch obtains appropriate global semantic correction while preserving local targets, effectively alleviating the semantic deficiency problem of non-representative image patches.

[0013] Optionally, cross-distillation losses can be determined as follows:

[0014]

[0015] in, For a collection of view pairs, For view pairs quantity; For the student view The set of locations of the occluded image patches, For the student view The number of shielding locations, For the first The mixed teacher objectives corresponding to each occlusion location It is the natural logarithm function. For student networks in student view The The proposed scheme achieves at least the following beneficial technical effects: by averaging all occlusion locations within a single student view and then averaging across all view pairs, it avoids gradient imbalance caused by differences in the number of occlusion locations between different student views, thus improving training stability; it accurately measures the difference between the student prediction and the mixed teacher objective at each occlusion location, enabling cross-distillation loss to directly drive the transfer of global semantics to image patch-level representations; and the image patch-level cross-distillation loss works synergistically with the image-level distillation loss to jointly optimize global semantic convergence and local detail preservation.

[0016] Optionally, the total loss may be determined as follows:

[0017]

[0018] in, For image-level distillation loss, For cross-distillation losses, These are the weighting coefficients for image-level distillation loss. The weighting coefficients are those for the cross-distillation loss. This scheme achieves at least the following beneficial technical effects: by flexibly balancing the contributions of image-level distillation loss and cross-distillation loss through weighting coefficients, it adapts to the different downstream tasks' emphasis on global semantics and dense representations; it enables the total loss to simultaneously supervise image-level [CLS] prediction and image patch-level occlusion prediction, improving the multi-scale representation learning ability of the student network; and through joint optimization, it enables the trained ViT encoder to possess both higher-quality image-level semantic aggregation capabilities and better image patch-level dense prediction capabilities.

[0019] According to a second aspect of the present invention, an image semantic segmentation method is provided, comprising: acquiring a trained image segmentation model, the image segmentation model including a feature extractor for extracting image patch embedding sequences from an input image and a segmentation predictor for upsampling the image patch embedding sequences to predict pixel-level semantics, wherein the feature extractor employs a ViT encoder of a student network or a ViT encoder of a teacher network trained as described in the first aspect; inputting an image to be segmented into the image segmentation model, the feature extractor extracting the image patch embedding sequences of the image to be segmented, and the segmentation predictor obtaining a segmentation result based on the image patch embedding sequences of the image to be segmented, the segmentation result including a pixel-level semantic prediction value for each pixel. This scheme can achieve at least the following beneficial technical effects: utilizing a ViT encoder trained through cross-distillation as a feature extractor, the image patch embeddings of the feature extractor containing global semantic correction information, improving the semantic segmentation model's ability to discriminate non-representative regions (such as background and edges); the segmentation predictor receiving high-quality image patch-level features, generating more accurate and continuous pixel-level semantic prediction results; and eliminating the need to train a large-scale ViT encoder from scratch, reducing the training cost and data annotation dependency of the semantic segmentation model, and improving segmentation performance.

[0020] According to a third aspect of the present invention, an image classification method is provided, comprising: acquiring a trained image classification model, the image classification model including a feature extractor for extracting image-level semantic features from an input image and a classification predictor for performing classification prediction based on the image-level semantic features, wherein the feature extractor employs a ViT encoder of a student network or a ViT encoder of a teacher network trained as described in the first aspect, and the image-level semantic features are [CLS] embeddings output by the feature extractor; inputting an image to be classified into the image classification model for classification, wherein the feature extractor extracts the [CLS] embeddings of the image to be classified, and the classification predictor obtains an image classification result based on the [CLS] embeddings of the image to be classified. This scheme can achieve at least the following beneficial technical effects: it utilizes the [CLS] embedding output by the ViT encoder trained by cross-distillation as image-level semantic features, and the [CLS] embedding aggregates the discriminative information after global semantic correction, thereby improving the accuracy of image classification; the feature extractor has a stronger semantic perception ability for low-attention regions, enhancing the robustness of the classification model in complex scenarios such as occlusion and deformation; it can quickly build high-performance image classification models, reduce the training overhead of downstream tasks, and improve image classification performance.

[0021] According to a fourth aspect of the present invention, a computer apparatus / device / system is provided, including a memory, a processor, and a computer program / instructions stored in the memory, wherein the processor executes the computer program / instructions to implement the steps of the method described in the first or second aspect.

[0022] According to a fifth aspect of the present invention, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first or second aspect. Attached Figure Description

[0023] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0024] Figure 1 This is a schematic diagram used to illustrate the phenomenon of representative image patches;

[0025] Figure 2 This is a flowchart illustrating the self-supervised training method of the ViT encoder according to an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of the data flow process of the self-supervised training method of the ViT encoder according to an embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram illustrating the performance gain of CODI on different [CLS] attention-ranked image blocks according to an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0029] As mentioned in the background section, existing technologies have at least the following shortcomings: First, image-level self-distillation signals are mainly transmitted via [CLS] attention readout, resulting in weak global semantic gradients obtained from low-attention image patches; second, ordinary occluded image patch distillation primarily learns local image patch targets, lacking direct global semantic correction; third, addressing this issue by changing the backbone structure or introducing manual annotations would increase training costs and weaken the versatility of self-supervised training. Therefore, there is an urgent need for a technical solution that injects global semantic information into each occluded image patch without changing the standard self-distillation framework or using manual labels.

[0030] To address these issues, the method of this invention employs the following means:

[0031] First, addressing the issue of insufficient supervision of low-attention image patches due to [CLS] attention gating of image-level semantic gradients, the method of this invention, in the image patch-level distillation path, for the location of the occluded image patch (the occluded location is obtained by random sampling, which may contain non-representative image patches with low attention), convexly combines the teacher image patch target distribution and the teacher [CLS] target distribution at the corresponding location according to a preset ratio to obtain a hybrid teacher target. This hybrid teacher target applies global semantic supervision to all occluded locations without explicitly relating it to their original [CLS] attention weights, enabling occluded image patches to obtain global semantic correction without passing through the [CLS] attention readout path. This overcomes the gating limitation of attention quality on semantic gradient propagation and improves the embedding (representation) quality of low-attention locations.

[0032] Secondly, addressing the issue that ordinary occluded image patch distillation only learns local targets and lacks global semantic correction, the method of this invention utilizes the hybrid teacher target as the distillation target for the student image patch prediction distribution at the occluded location. Since the hybrid teacher target simultaneously contains the local semantic components of the teacher image patch target distribution and the global semantic components of the teacher [CLS] target distribution, the occluded image patch not only learns local details during prediction and reconstruction but also simultaneously absorbs image-level global semantics, effectively alleviating the problems of insufficient semantics in non-representative image patches and the concentration of spatial prediction errors.

[0033] Finally, to address the issue that changing the backbone structure or introducing manual annotations would increase training costs and weaken generality, the method of this invention achieves global semantic injection only through convex combination operations at the target end without changing the standard self-distillation framework; it still uses unlabeled images for training, without the need for manual labels or modification of the ViT backbone structure, thereby significantly improving the quality of image block-level dense representation (i.e., embedding of all image blocks) while maintaining the generality and low cost of self-supervised training.

[0034] The present invention will be further described below from several aspects. Before specifically describing the embodiments of the present invention, the key findings obtained by the inventors through experimental observation and theoretical analysis during the research on the existing self-supervised visual Transformer training mechanism will be introduced first.

[0035] The findings not only reveal the underlying mechanism of uneven transmission of image-level semantics to dense image patch representations in existing technologies, but also directly constitute the technical motivation and theoretical basis for the subsequent proposal of the cross-distillation algorithm. Specifically, the inventors first quantitatively discovered the representative image patch phenomenon in self-supervised ViT through attention distribution analysis and dense prediction error statistics (corresponding to section I, Discovery and Quantification of Representative Image Patch Phenomena). Then, by establishing a local [CLS] attention readout model and deriving the gradient backpropagation form, they theoretically elucidated the [CLS] attention gating effect (corresponding to section II, Theoretical Analysis of [CLS] Attention Gating). Based on the above findings and theoretical explanations, the inventors ultimately proposed a cross-distillation scheme for occluded image patches (corresponding to section III, Training Method), and several downstream applications implemented based on the trained ViT encoder (corresponding to section IV, Application Scenarios). Finally, the inventors further verified the effectiveness of the method through experiments (corresponding to section V, Comparative Experiments).

[0036] I. Discovering and quantifying the phenomenon of representative image patches

[0037] By analyzing the attention distribution from [CLS] tags to image patch tags, dense prediction error locations, and error rates in different attention ranking regions, the inventors determined that the image-level semantic supervision obtained from low-attention image patches in existing self-distillation training is insufficient.

[0038] Specifically, the analysis steps include:

[0039] A1. Input unlabeled training images and use a self-supervised ViT encoder to extract the attention quality of the final layer [CLS] labels to each image patch. Define the top r% of image patches in the same image according to their attention quality as representative image patches, and define the remaining low-attention image patches as non-representative image patches.

[0040] A2. Freeze the pre-trained model and train a linear dense prediction head, then calculate the prediction error rate of pixels or image patches within different attention ranking intervals. See also Figure 1 The left curve shows the relationship between the number of pretraining epochs, VOCmIoU, and [CLS]-to-patch attention mass. The right side visualizes and compares the input image, attention map, prediction, and label at 500 epochs and 800 epochs. It can be seen that excessive attention leads to an increase in prediction errors. Experimental observations show that the error rate is higher for low-attention image patches, indicating that [CLS] labeling focusing on a small number of representative regions affects the spatial integrity of dense representations.

[0041] A3. Analyze the consistency of representative image patches across different instances. For categories such as animals, vehicles, and transportation, the model often repeatedly focuses on local discrimination regions such as eyes, mouths, wheels, and front of the vehicle; these regions are sufficient to support image-level recognition, but cannot cover the complete foreground region required for dense prediction. Therefore, the phenomenon of representative image patches is not an accidental result of a single image, but rather a systematic bias present in the self-supervised ViT image-level readout path.

[0042] Based on the above observations, the problem to be solved is: how to provide global semantic supervision for low-attention, unrepresentative image patches that is independent of [CLS] attention quality, so that they have more reliable semantic separability in dense prediction tasks.

[0043] II. Establishment and Analysis of the [CLS] Attention Gating Theory

[0044] A local [CLS] attention readout model is established, and the gradient form of the image-level self-distillation loss backpropagated to the image patch along the [CLS] readout path is derived, demonstrating that the semantic gradient received by the low-attention image patch is scaled by the attention quality. This theoretical analysis includes:

[0045] B1. Establish a local [CLS] attention readout model. Let the first... The key vectors of the image patches are The value vector is The [CLS] query vector is Attention score Attention quality Read the result with [CLS] They are respectively:

[0046]

[0047] in, For the first Attention score for each image patch. Transpose of the [CLS] query vector , For the first Key vectors for each image patch. , Dimension is the dimension of the key vector. For the first Attention quality of each image patch (Attention weight / Attention quality) It is an exponential function. For the first Attention score for each image patch. The [CLS] readout result is the output after the [CLS] tags are aggregated through an attention mechanism using all image patch value vectors. , For the first The attention quality (attention weight) of each image patch, and Same definition, only different index. For the first The value vector of each image patch , This represents the total number of image blocks.

[0048] B2. Let the teacher's [CLS] target distribution be... Students [CLS] predicted by Given that the image-level self-distillation loss can be expressed as:

[0049]

[0050] in, For the students' temperature, The global semantic gradient is backpropagated from the image-level self-distillation loss to the [CLS] readout result.

[0051] in, This is the image-level self-distillation loss. Let cross-entropy be the loss function. For the teacher [CLS] target distribution, For the Softmax function, For the student projection head, feature vectors are mapped to a K-dimensional prototype space. Read the results from [CLS] for the student projector head. The mapping output, ∈ , For student temperature parameters, >0, this temperature parameter is used to control the sharpening / smoothing level of the student output distribution; For global semantic gradient, That is, the gradient of the image-level loss with respect to the [CLS] readout result; To Find the gradient.

[0052] B3. Taking the derivative of the local attention readout yields the gradient forms of the image patch value path and the attention score path:

[0053]

[0054] in, For the first Value vector of image patches Find the gradient. For image-level self-distillation loss, For the first Attention quality of each image patch For global semantic gradient, To find the partial derivative, For the first Attention score for each image patch For the first The semantic utility of an image patch in local [CLS] readout For the first The value vector of each image patch The above formula illustrates that when the image-level semantic gradient is passed to the image patch along the [CLS] readout path, it is influenced by the attention quality. Scaling.

[0055] Therefore, it can be seen that highly attentioned representative image patches can obtain stronger image-level semantic updates, while low-attention, non-representative image patches obtain weaker image-level semantic updates. To overcome this gating effect, it is necessary to directly introduce the teacher [CLS] global semantics into the distillation target of the occluded image patches, so that each occluded image patch obtains a more comprehensive semantic update than the target image patch. Unrelated semantic correction.

[0056] III. Training Methods

[0057] According to one embodiment of the present invention, see Figure 2 This invention provides a self-supervised training method for a ViT encoder, comprising steps S1, S2, S3, S4, S5, and S6. To better understand this invention, each step is described in detail below with reference to specific embodiments.

[0058] Step S1: Perform augmentation sampling on all training images in the current batch to generate teacher views and student views and form view pairs. Sample the location set of occluded image patches for at least some student views, wherein the two views in each view pair correspond to the same spatial region of the same training image and use the same geometric transformation during augmentation sampling.

[0059] According to an embodiment of the present invention, in step S1, view pairs are generated as follows: For each training image in the current batch, a cropping region is randomly determined, and the cropping region is extracted from the training image and subjected to geometric transformations (horizontal flip, vertical flip, rotation, etc.) to obtain an initial view. The initial view is then copied twice to obtain a view pair consisting of a teacher view and a student view. Alternatively, further transformations can be performed on the initial view. For example, after copying the initial view twice, each copy is subjected to a random non-geometric visual transformation (such as color jitter, Gaussian blur, grayscale conversion, contrast adjustment) to obtain a view pair consisting of a teacher view and a student view. It should be noted that the cropping coordinates of the two views in this view pair are exactly the same, and the geometric transformations after cropping are the same, but the non-geometric visual transformations are randomly selected for each view. By repeating the above steps, a set of view pairs containing multiple view pairs can be obtained. All the teacher images constitute the teacher view set. All student images constitute the student view set. Subsequently, the view is divided into N image blocks, and the locations of the occluded image blocks are randomly sampled for the student view according to the occlusion ratio. To illustrate, assuming N = 14 × 14 and the occlusion ratio is set to 50%, 98 positions are randomly selected from 196 image patches, and the corresponding image patches are replaced with mask markers. The teacher view is not occluded. It should be understood that since the teacher view and student view in each view pair come from the same region of the same image and undergo the same geometric transformation, their pixel positions are in one-to-one correspondence. During subsequent self-supervised learning, the corresponding position of the occluded position of the student view in the teacher view can be determined according to the preset position mapping.

[0060] Step S2: Input the teacher view into the teacher network, extract the teacher [CLS] embedding and teacher image patch embedding sequence from the input view through its ViT encoder, and map the teacher [CLS] embedding to the teacher [CLS] target distribution through its projection head, and map the teacher image patch embedding sequence to the target distribution of each teacher image patch.

[0061] According to one embodiment of the present invention, the teacher network includes a ViT encoder and a projection head, which, for distinction, can be referred to as the teacher ViT encoder and the teacher projection head. The teacher network can be represented as:

[0062]

[0063] in, For teachers, ViT encoder, For teachers' projector heads.

[0064] For each teacher view The teacher encoder outputs the teacher [CLS] embedded information. and the Embedded teacher image blocks After processing with the teacher projection head, branch centering, and temperature sharpening, the teacher [CLS] target distribution was obtained. Teacher image patch target distribution The teacher [CLS] target distribution was obtained as follows:

[0065]

[0066] in, For the Softmax function, For teachers' projector heads, For Teacher Networks in Teacher View The output is the Teacher [CLS] embedding. The [CLS] branch of the teacher's projector is the operating center (used for decentralized stable training). , The teacher temperature parameter controls the degree of sharpening / smoothing in the teacher's output distribution. >0.

[0067] The target distribution of teacher image patches was obtained in the following manner:

[0068]

[0069] in, For the Softmax function, For teachers' projector heads, For Teacher Networks in Teacher View The output of the first Teacher patch embedding. The central point for the operation of the image block branches of the teacher's projector head. , For teachers' temperature parameters.

[0070] Because the two types of target distributions are mapped to the same object by the same projection head. The prototype space is 3D, so the two can be distributed and mixed.

[0071] Step S3: After masking the corresponding image patches according to their location set, the student view is input into the student network. The ViT encoder extracts the student [CLS] embedding and student image patch embedding sequence from the input view. The projector maps the student [CLS] embedding to the student [CLS] prediction distribution and the student image patch embedding sequence to the prediction distribution of each student image patch.

[0072] According to one embodiment of the present invention, the student network includes a ViT encoder and a projection head, which, for distinction, can be referred to as the student ViT encoder and the student projection head. The student network can be represented as:

[0073]

[0074] in, For students, ViT encoder, Projector head for students.

[0075] Student View Set by its location After masking the corresponding image patches, the images are input into the student network, and the student [CLS] embeddings are extracted by the student ViT encoder. And the student image patch embedding sequence. Wherein, corresponding to the set of occlusion locations. The image patch is replaced with a mask marker. The student ViT encoder predicts the position corresponding to the mask marker and outputs the embedding of the i-th student image patch. After processing with the student projector head and temperature sharpening, the predicted distribution of the student [CLS] was obtained. Student image patch prediction distribution at occluded locations .

[0076] The predicted distribution of students [CLS] is obtained as follows:

[0077]

[0078] Where softmax is the Softmax function. For student projector heads, For student networks in student view The student [CLS] embedding is output above. The student's temperature parameter controls the sharpening / smoothing level of the student's output distribution. >0.

[0079] The predicted distribution of student image patches at the occlusion locations is obtained as follows:

[0080]

[0081] in, For the Softmax function, For student projector heads, For student networks in student view The output of the first Student image patch embedding, This refers to the student's temperature parameter. For student view Set of occluded image patch locations The occlusion location index in the image patch prediction distribution is output only for the occlusion location and participates in subsequent loss calculation.

[0082] Since the student network needs to update its parameters through gradient descent, both the student [CLS] prediction distribution and the student image patch prediction distribution at the occluded location participate in backpropagation, driving the student network to approximate the target distribution output by the teacher network.

[0083] It should be noted that the student network and the teacher network above have the same neural network structure (isomorphic model), but the weight parameters of the network can be different.

[0084] Step S4: For each occluded position in the location set, the teacher image block target distribution and the teacher [CLS] target distribution at the corresponding position are convexly combined according to a preset ratio to obtain the mixed teacher target at the occluded position.

[0085] According to one embodiment of the present invention, a hybrid teacher target is determined as follows: A first weighting coefficient set for the teacher [CLS] target distribution and a second weighting coefficient set for the teacher image patch target distribution are obtained, wherein the sum of the first and second weighting coefficients is 1 and both are positive values; the first weighting coefficient is multiplied by the teacher [CLS] target distribution to obtain a global target component; the second weighting coefficient is multiplied by the teacher image patch target distribution corresponding to the occluded position in the teacher image patch target distribution to obtain a local target component; and the hybrid teacher target at the occluded position is obtained based on the global target component and the local target component. Preferably, the hybrid teacher target is determined as follows:

[0086]

[0087] in, For the first The mixed teacher objectives corresponding to each occlusion location The first weighting coefficient, The value range of is (0,1). The difference obtained is the second weighting coefficient. For Teacher View The determined first The distribution of teacher image patch targets at each occlusion location. For the teacher view The teacher [CLS] target distribution, The index of the occlusion location in the location set. When a smaller positive value is taken, each occluded image patch target contains a small amount of global [CLS] semantics, preferably, The value range is [0.05, 0.2], and preferably, .

[0088] Step S5: Determine the image-level distillation loss based on the student [CLS] prediction distribution and the teacher [CLS] target distribution in the view pair; determine the cross-distillation loss based on the student image patch prediction distribution at the occluded position and the mixed teacher target at the corresponding occluded position in the view pair; and obtain the total loss based on the image-level distillation loss and the cross-distillation loss.

[0089] According to an optional embodiment of the present invention, the total loss is determined in the following manner:

[0090]

[0091] in, For image-level distillation loss, For cross-distillation losses, These are the weighting coefficients for image-level distillation loss. This is the weighting coefficient for cross-distillation losses.

[0092] Image-level distillation loss is still determined using the standard [CLS] self-distillation loss, as follows:

[0093]

[0094] in, For student view Teacher View A view pair is formed. For a collection of view pairs, For transpose, It is the natural logarithm function. For student networks in student view The [CLS] predicted distribution output above.

[0095] Cross-distillation losses are determined as follows:

[0096]

[0097] in, For a collection of view pairs, For view pairs quantity; For the student view The set of locations of the occluded image patches, For the student view The number of shielding locations, For the first The mixed teacher objectives corresponding to each occlusion location It is the natural logarithm function. For student networks in student view The The predicted distribution of image patches output at each occlusion location.

[0098] According to another optional embodiment of the present invention, in addition to the two sub-loss terms described above, an optional auxiliary regularization sub-loss term may be added. For example, the total loss is determined as follows:

[0099]

[0100] in, For image-level distillation loss, For cross-distillation losses, To assist in regularization loss, These are the weighting coefficients for image-level distillation loss. The weighting coefficient for cross-distillation losses. These are the weight coefficients for the auxiliary regularization loss. The auxiliary regularization loss can be entropy regularization, used to impose entropy constraints on the output distribution of the student network to prevent pattern collapse and maintain prediction diversity; or it can be a contrastive loss, such as InfoNCE, used to bring the embedding distances of different views of the same training image (sample) closer together and widen the embedding distances of views of different training images to enhance embedding discriminativity. It should be understood that implementers can also use other forms of auxiliary regularization loss to further improve the performance of the ViT encoder.

[0101] Step S6: Update the ViT encoder and projector parameters of the student network according to the total loss, and update the ViT encoder and projector parameters of the teacher network by exponential moving average.

[0102] According to one embodiment of the present invention, the student network parameters are denoted as follows: It includes all trainable parameters of the student ViT encoder and student projector; the teacher network parameters are denoted as... It contains all the parameters of the teacher's ViT encoder and teacher's projector head.

[0103] In each training iteration, first, based on the total loss... Student network parameters Calculate the gradient and update the parameters using stochastic gradient descent:

[0104]

[0105] in, For learning rate, The total loss Student network parameters The gradient.

[0106] The teacher network does not backpropagate gradients; its parameters are updated by the student network using exponential moving averages (EMA).

[0107]

[0108] in, For EMA momentum, For teacher network parameters, For student network parameters.

[0109] Repeat steps S1 to S6 until training ends (e.g., network convergence or the number of training rounds reaches a preset value). The ViT encoders of the student network and the teacher network obtained after training can be used for downstream tasks.

[0110] According to an example of the invention, see Figure 3 This illustrates the complete forward propagation and target mixing process of the cross-distillation method. (Illustrative for the input view) First, sample the set of locations of the masked image patches. (As shown in the example, the actual value can be random), the image block at the corresponding position is replaced with the masking (mask) mark to obtain the masked student view; at the same time, the teacher view in the same cropping area is not masked.

[0111] Input the student view into the student network via the student ViT encoder The student [CLS] embedding and student image patch embedding sequence were extracted and processed by the student projection head. After mapping and softmax processing, the predicted distributions of students [CLS] are output respectively. Student image patch prediction distribution .

[0112] Input the teacher view into the teacher network via the teacher ViT encoder The teacher [CLS] embedding and teacher image patch embedding sequence are extracted and processed by the teacher projection head. The mapping process involves decentralization and softmax operations, which output the target distribution of teachers [CLS]. Teacher image patch target distribution .

[0113] During the cross-distillation stage, for each occlusion location (e.g., position) in the set of occlusion image patch locations... The teacher image patch target distribution at the corresponding location will be determined. Distribution of Teacher [CLS] Objectives According to proportion Perform weighted summation To obtain the mixed teacher target of the occlusion position .

[0114] Finally, the student [CLS] predicted distribution is determined. Distribution of Teacher [CLS] Objectives Determine image-level distillation loss Predicting distribution based on student image patches at occlusion locations Mixed teacher objectives with corresponding positions Determine the cross-distillation loss. Combine the image-level distillation loss with the cross-distillation loss to obtain the total loss. Update the student network parameters with minimizing the total loss as the optimization objective. The teacher network is not updated via gradient backpropagation; its parameters are updated by the student network via exponential moving average (EMA).

[0115] IV. Application Scenarios

[0116] After training, the ViT encoder of either the trained student network or teacher network is selected as the visual representation extractor for downstream tasks such as semantic segmentation, depth estimation, video object segmentation, and image classification. The following illustrations use image segmentation and image classification models as examples.

[0117] According to an embodiment of the present invention, an image semantic segmentation method is provided, comprising: obtaining a trained image segmentation model, the image segmentation model including a feature extractor for extracting image patch embedding sequences from an input image and a segmentation predictor for upsampling the image patch embedding sequences to predict pixel-level semantics, wherein the feature extractor is a ViT encoder of a student network or a ViT encoder of a teacher network trained using the self-supervised training method of the ViT encoder of the aforementioned embodiment; inputting an image to be segmented into the image segmentation model, wherein the feature extractor extracts the image patch embedding sequences of the image to be segmented, and the segmentation predictor obtains a segmentation result based on the image patch embedding sequences of the image to be segmented, the segmentation result including a pixel-level semantic prediction value for each pixel.

[0118] According to an embodiment of the present invention, an image classification method is provided, comprising: obtaining a trained image classification model, the image classification model including a feature extractor for extracting image-level semantic features from an input image and a classification predictor for performing classification prediction based on the image-level semantic features, wherein the feature extractor is a ViT encoder of a student network or a ViT encoder of a teacher network trained using the self-supervised training method of the ViT encoder of the aforementioned embodiment, and the image-level semantic features are the [CLS] embeddings output by the feature extractor; inputting an image to be classified into the image classification model for classification, wherein the feature extractor extracts the [CLS] embeddings of the image to be classified, and the classification predictor obtains the image classification result based on the [CLS] embeddings of the image to be classified.

[0119] V. Comparative Experiment

[0120] To verify the effectiveness of the method of the present invention, the inventors also conducted comparative experiments.

[0121] To comprehensively evaluate the improvement effect of the proposed method (CODI) on the representation quality of self-supervised visual Transformers, experiments were conducted within the self-distillation framework, using various backbone networks such as ViT-S / 14, ViT-B / 14, ViT-L / 14, and ViT-B / 16, for unsupervised pre-training on the ImageNet-1K training set. Currently mainstream image-level and image-block-level self-distillation methods such as DINOv2, DINO, iBOT, CAE, and Mugs were selected as baselines for comparison. Systematic transfer performance evaluations were performed on downstream tasks including linear semantic segmentation (COCO, ADE20K, PASCALVOC 2012), depth estimation (NYU Depth v2, KITTI, SUN RGB-D), video object segmentation (DAVIS 2017), and image classification (ImageNet-1K linear evaluation and k-NN evaluation). Tables 1 to 5 summarize the performance comparisons and ablation results of each method on different downstream tasks.

[0122] In the table below, the relevant indicators are listed below. This means the larger the value, the better. This indicates that smaller values ​​are better. DINOv2 The result represents the DINOv2 result reproduced under the alignment training settings; CODI represents the result obtained using the method of this invention.

[0123] Table 1. Results of Linear Semantic Segmentation and ImageNet-1K Classification

[0124]

[0125] As shown in Table 1, under the same data and similar training settings, CODI improves the average mIoU of semantic segmentation across small, medium, and large ViT scales, while also improving the Top-1 accuracy of frozen backbone image classification. This result demonstrates that the method of this invention not only improves dense representations but also does not sacrifice image-level representations.

[0126] Table 2 Depth Estimation Results

[0127]

[0128] As shown in Table 2, in the depth estimation task, CODI outperforms or is no less than Alignment DINOv2 on most metrics and achieves a better overall ranking across different model sizes. This indicates that the method of this invention improves not just a single semantic segmentation task, but a more general spatial representation quality.

[0129] Table 3. Results of semi-supervised video object segmentation using DAVIS 2017

[0130]

[0131] As shown in Table 3, CODI achieves higher performance on video target segmentation tasks based on representation similarity propagation. The metrics demonstrate that the method of this invention enables the image block embeddings (representations) learned by the ViT encoder to be more suitable for cross-frame pixel-level correspondence.

[0132] Table 4. CODI Mixing Ratio and Target Distribution Ablation Experiment

[0133]

[0134] Table 4 shows that mixing in a small number of teacher [CLS] target distributions can improve performance, while simply mixing in a uniform distribution will lead to a performance decrease; when When the value is too large, the semantics of image patches may be over-globalized and cause confusion. Therefore, the method of the present invention preferably uses a smaller positive mixing ratio, for example... .

[0135] Table 5. Results of the characterization distance analysis

[0136]

[0137] As shown in Table 5, although CODI reduces both intra-class and inter-class distances, it increases the inter-class-intra-class interval, indicating that the method of the present invention improves the class separability of image patch embedding.

[0138] As can be seen, the method of the present invention (CODI) outperforms the baseline method in various dense prediction tasks, verifying the effectiveness of improving the quality of dense representation by injecting global semantic information into occluded image patches.

[0139] In addition, by Figure 4 Furthermore, it can be seen that after sorting image patches by [CLS] attention quality from low to high, the performance gain of non-representative regions is significantly higher than that of representative regions. Specifically, the performance gain of image patches with the lowest attention ranking exceeds 4%, and as the attention ranking increases, the gain rapidly decreases and tends to plateau, while the gain of representative regions remains at around 2.3%. This further indicates that in existing self-distillation methods, non-representative image patches with low attention have greater potential for improvement in dense prediction performance due to the weaker semantic gradients obtained through the [CLS] attention readout path, further verifying the severity of the semantic imbalance problem.

[0140] In summary, existing methods primarily match local teacher targets for occluded image patches, but image-level semantics are subject to attention gating when transmitted via the [CLS] attention path, leading to semantic deficiencies in non-representative image patches. This invention injects a small amount of teacher [CLS] semantic distribution into each occluded image patch through cross-distillation of the target, improving the quality of image patch-level embedding without using downstream labels or altering the backbone structure. Example results demonstrate that the ViT encoder trained using this method improves transfer performance in downstream tasks such as semantic segmentation, depth estimation, video object segmentation, and image classification, while also enhancing model robustness even when representative image patches are occluded.

[0141] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0142] This invention can be a system, method, electronic device, computing device, computer program product and / or computer-readable medium.

[0143] Computer program products mainly refer to software products that implement various aspects of the present invention through computer programs, or hardware products that carry software that implements various aspects of the present invention.

[0144] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0145] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A self-supervised training method for a ViT encoder, comprising the following steps: S1. Perform augmented sampling on all training images in the current batch to generate teacher and student views and form view pairs. Sample the location set of occluded image patches for at least some of the student views, where... In each view pair, the two views correspond to the same spatial region of the same training image and use the same geometric transformation during augmentation sampling; S2. Input the teacher view into the teacher network, extract the teacher [CLS] embedding and teacher image block embedding sequence from the input view through its ViT encoder, and map the teacher [CLS] embedding to the teacher [CLS] target distribution through its projection head, and map the teacher image block embedding sequence to the target distribution of each teacher image block; S3. After masking the corresponding image blocks according to their position set, the student view is input into the student network. The ViT encoder extracts the student [CLS] embedding and student image block embedding sequence from the input view. The projector maps the student [CLS] embedding to the student [CLS] prediction distribution and the student image block embedding sequence to the prediction distribution of each student image block. S4. For each occluded position in the location set, the teacher image block target distribution and the teacher [CLS] target distribution at the corresponding position are convexly combined according to a preset ratio to obtain the mixed teacher target at the occluded position; S5. Determine the image-level distillation loss based on the student [CLS] prediction distribution and the teacher [CLS] target distribution in the view pair. Determine the cross-distillation loss based on the student image patch prediction distribution at the occluded position and the mixed teacher target at the corresponding occluded position in the view pair. Obtain the total loss based on the image-level distillation loss and the cross-distillation loss. S6. Update the ViT encoder and projector parameters of the student network according to the total loss, and update the ViT encoder and projector parameters of the teacher network by exponential moving average.

2. The method according to claim 1, characterized in that, Determine blended teacher objectives in the following manner: Obtain the first weighting coefficient set for the target distribution of the teacher [CLS] and the second weighting coefficient set for the target distribution of the teacher image patch, wherein the sum of the first and second weighting coefficients is 1 and both are positive values; Multiply the first weighting coefficient by the teacher [CLS] target distribution to obtain the global target component, and multiply the second weighting coefficient by the teacher image block target distribution at the corresponding occluded position in the teacher image block target distribution to obtain the local target component; Based on the global target component and the local target component, the hybrid teacher target at the occlusion location is obtained.

3. The method according to claim 1 or 2, characterized in that, Determine blended teacher objectives in the following manner: in, For the first The mixed teacher objectives corresponding to each occlusion location The first weighting coefficient, The value range of is (0,1). The difference obtained is the second weighting coefficient. For Teacher View The determined first The distribution of teacher image patch targets at each occlusion location. For the teacher view The teacher [CLS] target distribution, The occlusion location index is the location in the set of locations.

4. The method according to claim 3, characterized in that, Determine the cross-distillation loss as follows: in, For a collection of view pairs, For view pairs quantity; For the student view The set of locations of the occluded image patches, For the student view The number of shielding locations, For the first The mixed teacher objectives corresponding to each occlusion location It is the natural logarithm function. For student networks in student view The The predicted distribution of image patches output at each occlusion location.

5. The method according to claim 4, characterized in that, The total loss shall be determined as follows: in, For image-level distillation loss, For cross-distillation losses, These are the weighting coefficients for image-level distillation loss. This is the weighting coefficient for cross-distillation losses.

6. An image semantic segmentation method, characterized in that, include: A trained image segmentation model is obtained, the image segmentation model comprising a feature extractor for extracting image patch embedding sequences from an input image and a segmentation predictor for upsampling the image patch embedding sequences to predict pixel-level semantics, wherein the feature extractor employs a ViT encoder of a student network or a ViT encoder of a teacher network trained by the method as described in any one of claims 1 to 5. The image to be segmented is input into the image segmentation model, where its feature extractor extracts the image patch embedding sequence of the image to be segmented, and its segmentation predictor obtains the segmentation result based on the image patch embedding sequence of the image to be segmented. The segmentation result includes the pixel-level semantic prediction value of each pixel.

7. An image classification method, characterized in that, include: A trained image classification model is obtained, the image classification model including a feature extractor for extracting image-level semantic features from an input image and a classification predictor for making classification predictions based on the image-level semantic features, wherein the feature extractor adopts a ViT encoder of a student network or a ViT encoder of a teacher network trained by the method as described in any one of claims 1 to 5, and the image-level semantic features are the [CLS] embeddings output by the feature extractor; The image to be classified is input into the image classification model for classification. Its feature extractor extracts the [CLS] embedding of the image to be classified, and its classification predictor obtains the image classification result based on the [CLS] embedding of the image to be classified.

8. A computer device / apparatus / system, comprising a memory, a processor, and a computer program / instructions stored in the memory, wherein the processor executes the computer program / instructions to implement the steps of the method according to any one of claims 1-7.

9. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program / instructions stored thereon, the computer program / instructions being executed by a processor to implement the steps of the method according to any one of claims 1-7.

Images