A method and apparatus for training a self-supervised algorithm based on content-aware and self-distillation structures

Abstract

This disclosure provides a method and apparatus for training a self-supervised algorithm based on a content-aware and self-distillation structure. The method includes: processing an image using a first enhancement method and a second enhancement method to obtain a first enhanced image and a second enhanced image, respectively; obtaining the energy map of the first enhanced image to obtain a first energy map; obtaining the energy map of the second enhanced image to obtain a second energy map; obtaining a lossless representation set and a lossy representation set based on the first enhanced image, the second enhanced image, the first energy map, and the second energy map; inputting the lossy representation set and the lossless representation set into a student network and a teacher network in a self-distillation structure, respectively; and obtaining a total loss function based on the output results. This self-supervised algorithm training method based on a content-aware and self-distillation structure proposes a contrastive self-supervised learning training method based on an invariant causal mechanism, optimizing current contrastive self-supervised learning methods so that the training method can better utilize unlabeled data to train the model.

Description

Technical Field

[0001] This disclosure belongs to the field of image vision technology, specifically relating to a self-supervised algorithm training method, apparatus, device, and medium based on content-aware and self-distillation structure. Background Technology

[0002] Supervised learning is a commonly used training method in computer vision, frequently employed in image processing tasks such as image classification, object detection, and instance segmentation. While it has achieved great success in the past, it heavily relies on expensive and rare human labels and is prone to generalization errors, spurious correlations, and adversarial attacks. Consequently, research on supervised learning has reached a bottleneck. Summary of the Invention

[0003] This disclosure aims to at least address one of the technical problems existing in the prior art, and to provide a method, apparatus, device and medium for training a self-supervised algorithm based on a content-aware and self-distillation structure.

[0004] One aspect of this disclosure provides a method for training a self-supervised algorithm based on a content-aware and self-distillation structure, the method comprising:

[0005] The image is processed using a first enhancement method to obtain a first enhanced image, and the image is processed using a second enhancement method to obtain a second enhanced image. The first enhancement method and the second enhancement method are different.

[0006] Obtain the energy map of the first enhancement map to obtain the first energy map; obtain the energy map of the second enhancement map to obtain the second energy map.

[0007] Based on the first enhancement map, the second enhancement map, the first energy map, and the second energy map, a lossless characterization set and a lossy characterization set are obtained;

[0008] The lossy representation set is input into the student network in the self-distillation structure, and the lossless representation set is input into the teacher network in the self-distillation structure. The total loss function is obtained based on the output results.

[0009] The model weights of the student network and the teacher network are updated according to the total loss function.

[0010] Optionally, obtaining the energy map of the first enhancement map to obtain the first energy map includes:

[0011] Obtain the gradient of the first enhancement map to obtain the first energy map;

[0012] The step of obtaining the energy map of the second enhancement map to obtain the second energy map includes:

[0013] Obtain the gradient of the second enhancement map to obtain the second energy map.

[0014] Optionally, obtaining the lossless characterization set and the lossy characterization set based on the first enhancement map, the second enhancement map, the first energy map, and the second energy map includes:

[0015] A data preprocessing result set is obtained based on the first enhanced map, the second enhanced map, the first energy map, and the second energy map;

[0016] The data preprocessing result set is divided into equal-scale blocks to obtain the block-based data preprocessing result set;

[0017] Perform a convolution operation on the preprocessed data set after segmentation to obtain a one-dimensional local embedding representation block set;

[0018] A one-dimensional masked local embedding representation block set is obtained based on the one-dimensional local embedding representation block set and the learnable mask representation;

[0019] A first set of classification placeholders is generated based on the one-dimensional mask local embedding representation block set, and a second set of classification placeholders is generated based on the one-dimensional local embedding representation block set.

[0020] The lossy representation set is obtained based on the one-dimensional mask local embedding representation block set and the first classification placeholder set, and the lossless representation set is obtained based on the one-dimensional local embedding representation block set and the second classification placeholder set.

[0021] Optionally, the step of inputting the lossy representation set into the student network in the distillation structure, inputting the lossless representation set into the teacher network in the distillation structure, and obtaining the total loss function based on the output results includes:

[0022] The lossy representation set is input into the student network in the distillation structure to obtain the information category prediction probability and the first semantic category probability.

[0023] The lossless representation set is input into the teacher network in the distillation structure to obtain the information category prediction probability and the second semantic category probability.

[0024] The information loss value is obtained based on the predicted probability of the information category and the estimated probability of the information category.

[0025] The semantic loss value is obtained based on the first semantic category probability and the second semantic category probability;

[0026] The total loss function is obtained based on the information loss value and the semantic loss value.

[0027] Optionally, the step of inputting the lossy representation set into the student network in the distillation structure to obtain the information category prediction probability and the first semantic category probability includes:

[0028] The one-dimensional local embedding representation block set and the first classification placeholder set in the lossy representation set are input into the encoder to obtain the student network output result;

[0029] The student network output and the learnable mask representation in the lossy representation set are input into the predictor to obtain the information category prediction probability and the first semantic category probability.

[0030] Optionally, the step of inputting the lossless representation set into the teacher network in the distillation structure to obtain the information category prediction probability and the second semantic category probability includes:

[0031] The one-dimensional local embedding representation block set and the second classification placeholder set in the lossless representation set are input into the encoder to obtain the teacher network output result;

[0032] The output of the teacher network is input into the predictor to obtain the estimated probability of the information category and the probability of the second semantic category.

[0033] Optionally, obtaining the total loss function based on the information loss value and the semantic loss value includes:

[0034] The total loss function is obtained by summing the information loss value and the semantic loss value.

[0035] Optionally, updating the model weights of the student network and the teacher network according to the total loss function includes:

[0036] The student network is backpropagated using the total loss function to update its model weights.

[0037] Optionally, updating the model weights of the student network and the teacher network according to the total loss function includes:

[0038] The updated model weights of the student network are processed using an exponential moving average, and the processed weights are used as the model weights of the teacher network.

[0039] Another aspect of this disclosure provides a self-supervised algorithm training device based on a content-aware and self-distillation structure, the device comprising:

[0040] An image enhancement module is used to process an image using a first enhancement method to obtain a first enhanced image, and to process the image using a second enhancement method to obtain a second enhanced image, wherein the first enhancement method and the second enhancement method are different;

[0041] An energy map acquisition module is used to acquire the energy map of the first enhancement map to obtain a first energy map, and to acquire the energy map of the second enhancement map to obtain a second energy map;

[0042] The characterization set acquisition module is used to obtain a lossless characterization set and a lossy characterization set based on the first enhancement map, the second enhancement map, the first energy map, and the second energy map.

[0043] The self-distillation module is used to input the lossy characterization set into the student network of the self-distillation structure, and the lossless characterization set into the teacher network of the self-distillation structure, and obtain the total loss function based on the output results;

[0044] An update module is used to update the model weights of the student network and the model weights of the teacher network according to the total loss function.

[0045] Another aspect of this disclosure provides an electronic device comprising:

[0046] One or more processors;

[0047] A storage unit is used to store one or more programs, which, when executed by one or more processors, enable the one or more processors to implement a self-supervised algorithm training method based on a content-aware and self-distillation structure as described above.

[0048] Another aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon.

[0049] When the computer program is executed by the processor, it can implement the self-supervised algorithm training method based on content awareness and self-distillation structure described above.

[0050] This disclosure discloses a self-supervised algorithm training method, apparatus, device, and medium based on content-aware and self-distillation structures. It proposes a contrastive self-supervised learning training method based on an invariant causal mechanism. Content-aware technology uses energy maps to quantify the importance of image content, expanding the reconstruction target from a single "reconstructed pixel" to "reconstructed energy," thereby improving the semantic level of the latent representations learned by the encoder and enhancing feature robustness. By introducing a self-distillation structure, online tokenizers are implemented, enabling one-step training and resolving the problem of information discrepancies between information extractor domains, reducing training complexity and time consumption. Combining content-aware technology and self-distillation structures, a causal mechanism is introduced to constrain the learning process, thus optimizing the current contrastive self-supervised learning method. This allows the training method to better utilize unlabeled data for model training. Models trained using this method can be adapted to other tasks in the image vision domain, including image classification, object detection, and instance segmentation, through targeted fine-tuning. Attached Figure Description

[0051] Figure 1 A schematic block diagram of an example electronic device for implementing a self-supervised algorithm training method and apparatus based on a content-aware and self-distillation structure according to an embodiment of the present disclosure;

[0052] Figure 2 This is a flowchart illustrating a self-supervised algorithm training method based on a content-aware and self-distillation structure, according to another embodiment of this disclosure.

[0053] Figure 3 This is a detailed flowchart of a self-supervised algorithm training method based on content-aware and self-distillation structure, according to another embodiment of this disclosure.

[0054] Figure 4 This is a flowchart illustrating a self-supervised algorithm training method based on content-aware and self-distillation structure, according to another embodiment of this disclosure.

[0055] Figure 5 This is a schematic diagram of a self-supervised algorithm training device based on a content-aware and self-distillation structure, according to another embodiment of this disclosure. Detailed Implementation

[0056] To enable those skilled in the art to better understand the technical solutions of this disclosure, the disclosure will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0057] Unless otherwise specifically stated, the technical or scientific terms used in this disclosure should be understood in their ordinary sense by one of ordinary skill in the art to which this disclosure pertains. The terms "comprising" or "including," as used in this disclosure, do not limit the shapes, numbers, steps, actions, operations, components, elements, and / or groups thereof mentioned, nor do they exclude the appearance or inclusion of one or more other different shapes, numbers, steps, actions, operations, components, elements, and / or groups thereof, or the inclusion of these. In the description of this disclosure, "a plurality of" means two or more, unless otherwise expressly specified.

[0058] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this disclosure. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale, and techniques, methods, and devices known to those skilled in the art may not be discussed in detail; however, where appropriate, the illustrated techniques, methods, and devices should be considered part of the specification. In all the examples shown and discussed herein, any other specific example may have different values. It should be noted that similar symbols and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0059] Before discussing in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when an operation is completed, but it may also have additional steps not included in the figures. The process may correspond to a method, function, procedure, subroutine, subroutine, etc.

[0060] First, refer to Figure 1 This describes an example electronic device for implementing a content-aware and self-distillation-based self-supervised algorithm training method and apparatus according to embodiments of the present disclosure.

[0061] like Figure 1 As shown, the electronic device 200 includes one or more processors 210, one or more storage devices 220, input devices 230, output devices 240, etc., and these components are interconnected via a bus system and / or other forms of connection mechanism 250. It should be noted that... Figure 1 The components and structures of the electronic device shown are merely exemplary and not limiting; the electronic device may also have other components and structures as needed.

[0062] For example, the processor can effectively perform a series of operations such as data access, data storage, data processing, and data analysis on large-scale real-time data from multiple heterogeneous sources.

[0063] The processor 210 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0064] The storage device 220 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and a processor may execute the program instructions to implement the client functions (implemented by the processor) in the embodiments of this disclosure described below, and / or other desired functions. Various applications and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the applications.

[0065] The input device 230 may be a device used by a user to input commands, and may include one or more of the following: keyboard, mouse, microphone, and touch screen.

[0066] The output device 240 can output various information (such as images or sounds) to the outside (e.g., a user) and may include one or more of a display, a speaker, etc.

[0067] For example, an example electronic device for implementing a self-supervised algorithm training method and apparatus based on a content-aware and self-distillation structure according to embodiments of the present disclosure can be implemented as a smartphone or tablet computer.

[0068] Supervised learning has achieved great success in the past, but it relies heavily on expensive and rare human labels and is prone to generalization errors, spurious correlations, and adversarial attacks, thus leading to a bottleneck in research on supervised learning. Recently, pre-training models based on self-supervised learning have made great progress in computer vision due to their good data utilization efficiency and generalization ability, and have formed a mainstream training paradigm called "self-supervised pre-training-task-based fine-tuning".

[0069] Currently, the mainstream model training method mainly employs supervised learning, the core idea of ​​which is to use labeled data to provide strong supervisory feedback to the model. During training, by calculating the loss function (i.e., strong supervisory feedback) and utilizing gradient backpropagation, the model continuously adjusts its parameters, gradually fitting the target output distribution. Therefore, supervised learning is a very intuitive and efficient training method, but its drawbacks are also quite obvious. It heavily relies on expensive and rare human labels; in most real-world scenarios, only a small portion of the massive amounts of data can be labeled, leaving a large amount of unlabeled data unusable. Furthermore, training with limited data can lead to poor generalization, poor robustness, and susceptibility to noise attacks, among other problems.

[0070] To address the various problems associated with supervised learning, the mainstream solution currently focuses on training methods based on self-supervised learning. Self-supervised learning can be divided into three technical directions: 1) Contrastive learning, which involves: firstly, constructing positive and negative examples from a batch of data through data augmentation (such as horizontal flipping, vertical flipping, contrast adjustment, cropping, etc.). The original samples and the augmented samples in this batch are considered positive examples, while other samples and samples augmented based on other samples are considered negative examples. Then, the positive and negative samples are input into two identical neural network models. These two models simultaneously output embedded representations of the samples. Finally, a specific loss function (such as calculating the distance between two embedded representations) is used to narrow the distance between positive samples and widen the distance between negative samples, enabling the neural network model to learn to extract sample representations without labels. 1) Generative learning replaces the pre-training stage that previously required large-scale labeled data; 2) Generative learning generally adopts an "encoder-decoder" structure. Its main approach is to use an encoder to encode the image input into an explicit representation vector, and then use a decoder to reconstruct the image using the explicit representation vector. Finally, by minimizing the reconstruction error, the model can learn explicit representation vectors containing potential information even without labels; 3) Generative contrastive learning (also known as adversarial learning) combines the previous two methods and adopts a "generator-discriminator" structure. The generator generally adopts an "encoder-decoder" structure, and the discriminator generally adopts a classifier structure. Its main approach is to first generate fake samples through the generator, and then simultaneously input real and fake samples into the discriminator to distinguish between real and fake samples.

[0071] Current mainstream self-supervised techniques primarily employ contrastive learning or generative learning. Among these, the mainstream algorithms in contrastive learning include the MoCo series, SimCLR, DINO, etc. These algorithms all suffer from the following problems:

[0072] 1) Both adopt a "Siamese network" training structure, meaning that each step in the training phase requires forward computation of two neural network models with identical structures (equivalent to training twice), thus consuming more computing power and time. In addition, different model parameter update strategies result in varying memory / GPU space consumption. In the worst case, if the parameters of both neural network models need to be updated, then the memory / GPU space consumption will double compared to supervised training.

[0073] 2) As discussed in Section 2, the contrastive learning model heavily relies on data augmentation techniques. To ensure model performance, different data augmentation techniques need to be carefully designed for different tasks. Therefore, model performance is limited by the design of data augmentation techniques. Furthermore, the knowledge learned by the model may only be a certain transformation characteristic, rather than truly useful information.

[0074] 3) Contrastive learning typically requires a large number of negative samples during training (at least 4096 samples), therefore, the memory / GPU space required is enormous.

[0075] Mainstream algorithms in generative learning include BEiT, MAE, and SimMIM. These algorithms employ an encoder-decoder structure. The encoder encodes x into an explicit representation vector z, and the decoder attempts to reconstruct the original input x' using the representation vector z. Finally, the encoder learns the latent representation of the data by minimizing the reconstruction difference. The network structures of the encoder and decoder do not need to be identical. For example, MAE uses an asymmetric encoder-decoder structure, where the encoder can be any Transformer model, and the decoder is a lightweight Transformer model. Furthermore, the reconstruction goals of these algorithms are pixel-level; the decoder aims to reconstruct the pixel value at a specific location in the original input using the representation vector z. Compared to contrastive learning, generative learning does not require a "Siamese network," does not rely on data augmentation techniques, and does not require a large number of negative samples. Therefore, it has significant advantages in terms of computational and space consumption, and model performance is no longer limited by any particular data augmentation technique. However, these generative self-supervised learning methods also have some problems:

[0076] 1) The reconstruction objective determines the semantic level of the latent representation learned by the encoder; in other words, it affects the information density of the latent representation, which in turn affects the performance of downstream tasks. The reconstruction objective of the mainstream algorithms mentioned above is pixel-level restoration, that is, restoring the pixel values ​​at the corresponding positions. However, pixels are data with very low information density and high redundancy. For example, removing some pixels from an image will not affect the semantics (content) of the entire image. Therefore, the "pixel reconstruction" objective actually provides limited information to the encoder, thus limiting the semantic level of the latent representation to some extent.

[0077] 2) The information carried by pixel values ​​is the content color and brightness, which are extremely susceptible to noise from environmental factors, such as changes in ambient light intensity. Therefore, the latent representation learned by the encoder is also affected by noise to some extent, i.e., its robustness is weak.

[0078] 3) Currently, mainstream generative self-supervised learning algorithms for computer vision are derived from NLP (Natural Language Processing) generative self-supervised learning algorithms. Therefore, the concept of tokens (word vectors) is also applied to some computer vision generative self-supervised learning algorithms. However, unlike NLP, where each word in natural language contains high-level semantic information and can be automatically converted into a fixed-size token (containing the semantic information of a specific word) through Word2Vec, and a complete vocabulary provides supervision signals during training, in computer vision, pixels contain redundant and low-level semantic information. Each token is essentially an embedded vector of a local pixel block, so each token contains little semantic information and lacks a corresponding vocabulary. Therefore, some mainstream computer vision generative self-supervised learning algorithms require a two-step process: first, a tokenizer is trained in advance, and then self-supervised training is performed. This paradigm is called offline tokenizer. Because the training process is divided into two steps, it can lead to problems such as information discrepancies between information extractor domains (the data used to train the information extractor and the data used when the information extractor is distributed very differently), high operational complexity, and long processing time.

[0079] Below, we will refer to Figure 2 and Figure 3 This disclosure describes a method for training a self-supervised algorithm based on a content-aware and self-distillation structure, according to an embodiment of the present disclosure.

[0080] like Figure 2 and Figure 3 As shown, a self-supervised algorithm training method based on content-aware and self-distillation structure includes:

[0081] S100: The image is processed using a first enhancement method to obtain a first enhanced image, and the image is processed using a second enhancement method to obtain a second enhanced image. The first enhancement method and the second enhancement method are different.

[0082] Specifically, in this step, the input data is an image I with a scale of H*W, where H and W can be selected according to the actual situation so that image I can be adjusted to different scales as needed. It should be noted that image I is... Figure 3 In this embodiment, the "original RGB image" is exemplarily selected as image I. In other embodiments, other images, such as grayscale images, can be selected according to the actual usage.

[0083] Specifically, in this step, image I is first processed using two arbitrary and different data augmentation methods, namely, augmentation method A0 and augmentation method A1, to obtain first augmented image I0 and second augmented image I1, respectively. It should be noted that the first augmented image I0 corresponds to... Figure 3 "Enhancement" in Figure 1 The second enhanced image I1 corresponds to Figure 3 "Enhancement" Figure 2 In this step, different data augmentation methods can be selected according to the actual situation, and this disclosure does not limit this approach.

[0084] S200: Obtain the energy map of the first enhancement map, obtain the first energy map, obtain the energy map of the second enhancement map, and obtain the second energy map.

[0085] Specifically, in this step, the energy calculation formula e2 is used to calculate the energy of each channel (R, G, B) for the first enhancement map I0 and the second enhancement map I1, respectively, resulting in the first energy map E0 and the second energy map E1. It should be noted that the first energy map E0 corresponds to... Figure 3 "energy" Figure 1 The second energy diagram E1 corresponds to Figure 3 "energy" Figure 2 The energy map can effectively reflect the important features of an image, and the magnitude of the energy is directly proportional to the importance of the content. In this step, different energy calculation formulas e2 can be selected according to the actual use, and this embodiment of the disclosure is not limited in this regard.

[0086] S300: Obtain the lossless characterization set and the lossy characterization set based on the first enhancement map, the second enhancement map, the first energy map, and the second energy map.

[0087] Specifically, in this step, different methods can be selected to obtain the lossless characterization set and the lossy characterization set according to the actual situation. For example, the lossless characterization set can be obtained based on the first enhancement map, the second enhancement map, the first energy map and the second energy map, and the lossy characterization set can be obtained based on the first enhancement map, the second enhancement map, the first energy map and the second energy map. Other methods can also be used, such as obtaining the lossless characterization set based on the first enhancement map, the second enhancement map, the first energy map and the second energy map, and obtaining the lossy characterization set based on the lossless characterization set.

[0088] S400: Input the lossy representation set into the student network in the distillation structure, input the lossless representation set into the teacher network in the distillation structure, and obtain the total loss function based on the output results.

[0089] Specifically, in this step, both the student network and the teacher network in the self-distillation structure use encoders and predictors. For example, the student network and teacher network use a twin network structure, i.e., student network f... student and teacher network f teacher The model structures are exactly the same. The encoders of the student network and the teacher network are arbitrary Transformer structure models, while the predictors are MLP layer models with a structure of "fully connected layer + nonlinear activation function + fully connected layer".

[0090] It should be noted that in this step, the structure of the student network and the teacher network can be selected according to the actual use case. This embodiment of the disclosure does not limit this, and the network structure given in this embodiment is only an example.

[0091] S500: Update the model weights of the student network and the teacher network according to the total loss function.

[0092] Specifically, in this step, different methods can be selected to update the model weights of the student network and the teacher network according to the actual situation to complete the training. This embodiment of the disclosure does not limit this.

[0093] This disclosure discloses a self-supervised algorithm training method based on content-aware and self-distillation structures, introducing two important core technologies: content-aware technology and self-distillation structures. Content-aware technology uses energy maps to quantify the importance of image content, expanding the reconstruction target from a single "reconstructed pixel" to "reconstructed energy," thereby improving the semantic level of the latent representations learned by the encoder and enhancing feature robustness. By introducing a self-distillation structure, online tokenizers are implemented, enabling one-step training and addressing the issue of information discrepancies between domains of the tokenizer, reducing training complexity and time consumption. Combining content-aware technology and self-distillation structures, a causal mechanism is introduced to constrain the learning process, thus optimizing current contrastive self-supervised learning methods. This allows the training method to better utilize unlabeled data for model training. Models trained in this way can be adapted to other tasks in the image vision domain, including image classification, object detection, and instance segmentation, through targeted fine-tuning.

[0094] The specific steps of step S200 in a self-supervised algorithm training method based on content awareness and self-distillation structure in the embodiment will be further described below.

[0095] For example, in step S200, obtaining the energy map of the first enhancement map and obtaining the first energy map includes: obtaining the gradient of the first enhancement map and obtaining the first energy map.

[0096] For example, in step S200, obtaining the energy map of the second enhancement map and obtaining the second energy map includes: obtaining the gradient of the second enhancement map and obtaining the second energy map.

[0097] Specifically, in this step, the energy calculation formula e2 is as follows:

[0098]

[0099]

[0100]

[0101]

[0102]

[0103]

[0104] in, These represent the gradients in the x-direction of each RGB channel. Let i represent the y-direction gradient of each RGB channel, where i takes the value 0 or 1. When i is 0, it corresponds to the first enhancement image I0 and the first energy image E0; when i is 1, it corresponds to the second enhancement image I1 and the second energy image E1. It should be noted that since an RGB image is selected as the input image in this embodiment, the gradients of the R, G, and B channels are obtained separately in this step. If a grayscale image is selected as the input image, it is not necessary to obtain the gradients of multiple channels.

[0105] The specific steps of step S300 in a self-supervised algorithm training method based on content awareness and self-distillation structure, as described in the following embodiment, will be further elaborated below.

[0106] For example, in step S300, obtaining the lossless characterization set and the lossy characterization set based on the first enhancement map, the second enhancement map, the first energy map, and the second energy map includes:

[0107] S310: Obtain the data preprocessing result set based on the first enhanced map, the second enhanced map, the first energy map, and the second energy map.

[0108] Specifically, in this step, after completing the energy map calculation, we can obtain the data preprocessing result set I. all ={I0,I1,E0,E1}.

[0109] S320: Divide the data preprocessing result set into equal-scale blocks to obtain the block-preprocessed data preprocessing result set.

[0110] Specifically, in this step, for I all Each data point is divided into equal-scale blocks. Each local block is n*m in size, then I... all The number of local blocks that can be obtained from each data point is After the blocks are divided, P is obtained one-to-one. all ={P I0 ,P I1 ,P E0 ,P E1 It should be noted that in this step, the values ​​of m and n can be selected according to the actual usage to select local blocks of different sizes, and this embodiment of the disclosure is not limited in this regard.

[0111] S330: Perform a convolution operation on the preprocessed data set after segmentation to obtain a set of one-dimensional local embedding representation blocks.

[0112] Specifically, in this step, for P I0 ,P I1 ,P E0 ,P E1 Using convolution kernels respectively convolution kernel convolution kernel Perform a convolution operation, where c0 is the number of channels in the encoder input. It should be noted that R represents the set of real numbers; that is, in this step, different convolution kernels can be selected from the real numbers depending on the actual application.

[0113] After the operation is completed, the results for P are obtained respectively. I0 ,P I1 ,P E0 ,P E1 The convolution result EBD I0 ,EBD I1 ,EBD E0 ,EBD E1 This leads to the one-dimensional local embedding representation block set EBD = {EBD} I0 ,EBD I1 ,EBD E0 ,EBD E1},in:

[0114]

[0115]

[0116]

[0117]

[0118] in, Let each be a one-dimensional local embedding representation block, k = 0, 2, ..., N, representing the number of one-dimensional local embedding representation blocks; j = {I0, I1, E0, E1}, where I0 corresponds to the output of the first enhancement map, I1 corresponds to the output of the second enhancement map, E0 corresponds to the output of the first energy map, and E1 corresponds to the output of the second energy map. The tensor shapes are all

[0119] It should be noted that the one-dimensional locally embedded representation block set EBD corresponds to Figure 3 The “lossless local embedding representation block set” in the middle, including EBD I0 EBD I1 EBD E0 EBD E1 and Figure 3 The block sets in the lossless local embedding representation block set correspond sequentially from left to right.

[0120] S340: Obtain the one-dimensional masked local embedding representation block set based on the one-dimensional local embedding representation block set and the learnable mask representation.

[0121] Specifically, in this step, a random masking operation is performed on the EBD, that is, a certain proportion is randomly selected from N one-dimensional local embedding representation blocks, and the selected one-dimensional local embedding representation blocks are represented by a learnable mask v. mask Perform a replacement (equivalent to "destroying" the lossless local embedding representation set), and then use a normal distribution. The parameters of the learnable mask representation are randomly initialized, ultimately resulting in a one-dimensional mask local embedding representation block set (MEBD), corresponding to... Figure 3 The “lossy local embedding representation block set” is as follows:

[0122] MEBD = {MEBD I0 MEBD I1 MEBD E0 MEBD E1}

[0123]

[0124]

[0125]

[0126]

[0127] in, One-dimensional local embedding mask representation block, m k ∈{0,1} N When m k When the value is 1, it represents a one-dimensional local embedding representation block at the corresponding position. Replaced with v mask Otherwise, it represents a one-dimensional locally embedded representation block. It remains unchanged.

[0128] S350: Generate the first classification placeholder set MEBD based on the one-dimensional mask local embedding representation block set. cls The second classification placeholder set EBD is generated based on the one-dimensional local embedding representation block set. cls .

[0129] Specifically, in this step, the first category placeholder set MEBD cls Second Class Placeholder Set EBD cls They are respectively:

[0130] MEBD cls ={mc I0 ,mc I1 ,mc E0 ,mc E1}

[0131] EBD cls ={c I0 ,c I1 ,c E0 ,c E1}

[0132] S360: Obtain a lossy representation set based on the one-dimensional mask local embedding representation block set and the first classification placeholder set, and obtain a lossless representation set based on the one-dimensional local embedding representation block set and the second classification placeholder set.

[0133] Specifically, in this step, based on the one-dimensional mask local embedding representation block set MEBD and the first classification placeholder set MEBD cls Obtain the lossy representation set:

[0134] X mask ={MEBD cls MEBD

[0135] Based on the one-dimensional locally embedded representation block set EBD and the second classification placeholder set EBD cls Obtain the lossless characterization set:

[0136] X good ={EBD cls ,EBD}

[0137] Specifically, in this step, other methods can also be used to construct lossy and lossless characterization sets, and this disclosure does not limit this approach.

[0138] The following will further elaborate on the specific steps of step S400 in a self-supervised algorithm training method based on content awareness and self-distillation structure in an embodiment.

[0139] For example, in step S400, the lossy representation set is input into the student network in the distillation structure, and the lossless representation set is input into the teacher network in the distillation structure. The total loss function is obtained based on the output, including:

[0140] S410: Input the lossy representation set into the student network in the distillation structure to obtain the information category prediction probability and the first semantic category probability. Specifically, this step includes:

[0141] S411: The lossy representation set X mask ={MEBD cls One-dimensional masked local embedding representation block set in MEBD; unreplaced local embedding representation blocks in MEBD and the first category placeholder set MEBD cls The input encoder is the set of one-dimensional locally embedded representation blocks and the set of first classification placeholders. After being processed by the transformer model, the output of the student network is obtained.

[0142] S412: Combine the student network output with the learnable masked representation v from the lossy representation set. mask Input predictor, combined with encoder output and learnable mask representation The MLP layer model predicts the information category prediction probability (which can indirectly represent pixel category or energy category) and the first semantic category probability at the "damaged" location. Therefore, the role of the student network is to indirectly reconstruct the original image information and the original image energy.

[0143] The obtained information category prediction probability is The probability of the first semantic category is Where j = {I0, I1, E0, E1}, I0 corresponds to the output of the first enhancement map, I1 corresponds to the output of the second enhancement map, E0 corresponds to the output of the first energy map, and E1 corresponds to the output of the second energy map. c out This represents the number of channels in the predictor output.

[0144] It should be noted that the information category in the information category prediction probability and the information category estimated probability can be determined according to the actual situation. For example, image pixel values ​​can be selected. Image pixel values ​​are discrete values ​​with a value range of 0 to 255, which means there are 256 information categories. The semantic category in the first semantic category probability and the second semantic category probability can be determined according to the actual situation. For example, implicit semantic information generated by the model is not limited in this embodiment.

[0145] Specifically, in this step, the outputs of the student network constitute a set:

[0146] Token mask ={token cls0 token pred}

[0147]

[0148]

[0149] Among them, token cls0 For the first semantic category probability set, corresponding to Figure 3 "Semantic category probability set 1", token pred For the information category prediction probability set, corresponding to Figure 3 The "information category prediction probability set" has the first semantic category probability as follows: Information category prediction probability is Where j = {I0, I1, E0, E1}, I0 is used to correspond to the output of the first enhancement map, I1 is used to correspond to the output of the second enhancement map, and E0 is used to correspond to the output of the first energy map.

[0150] S420: Input the lossless representation set into the teacher network in the distillation structure to obtain the information category prediction probability and the second semantic category probability. Specifically, this step includes:

[0151] S421: Transfer the lossless representation set X good ={EBD cls The set of one-dimensional local embedded representation blocks in EBD and the set of second-class placeholders in EBD. cls Input the encoder to obtain the teacher's network output.

[0152] S422: Input the output of the teacher network into the predictor to obtain the information category prediction probability and second semantic category probability of all local embedded representation blocks in the one-dimensional local embedded representation block set EBD in the lossless representation set.

[0153] Since the teacher network is lossless, its role is that of an information extractor (tokenizer). Its function is to obtain the estimated probability distribution of all local embedded representation blocks in EBD through the predictor, and use the estimated probability distribution as a supervision signal to supervise the learning of the student network. In other words, the probability distribution predicted by the predictor of the student network should be as close as possible to the estimated probability distribution of the teacher network.

[0154] The estimated probability of the obtained information category is The probability of the second semantic category is Where j = {I0, I1, E0, E1}, I0 corresponds to the output of the first enhancement map, I1 corresponds to the output of the second enhancement map, E0 corresponds to the output of the first energy map, and E1 corresponds to the output of the second energy map. c out This represents the number of channels in the predictor output.

[0155] Specifically, in this step, the outputs of the teacher network constitute a set:

[0156] Token good ={token cls1 token gt}

[0157]

[0158]

[0159] Among them, token cls1For the second semantic category probability set, corresponding to Figure 3 "Semantic category probability 2" in the token gt For the information category, predict the probability set, corresponding to Figure 3 The "information category prediction probability set" in the text.

[0160] S430: Obtain the information loss value L based on the predicted probability and estimated probability of information category. info .

[0161] S440: Obtain the semantic loss value L based on the first semantic category probability and the second semantic category probability. cls .

[0162] S450: Based on information loss value L info and semantic loss value L cls The total loss function L is obtained. total .

[0163] Specifically, in this step, based on the information loss value L... info and semantic loss value L cls The sum of these yields the total loss function L. total .

[0164] Among them, the semantic loss value L cls This ensures the semantic validity of "online information extraction," with an information loss value L. info This ensures that the difference between the information "learned" by the student network and the information "provided" by the teacher network is minimized. The specific definitions of both are as follows:

[0165]

[0166]

[0167] L total =L cls +L info

[0168] It should be noted that the order of steps S410 and S420 is not restricted; they can be executed in different orders or simultaneously. Similarly, the order of steps S430 and S440 is not restricted; they can be executed in different orders or simultaneously.

[0169] The specific steps of step S500 in a self-supervised algorithm training method based on content awareness and self-distillation structure in the embodiment will be further described below.

[0170] For example, in step S500, updating the model weights of the student network and the teacher network according to the total loss function includes:

[0171] S510: Perform gradient backpropagation on the student network based on the total loss function to update the model weights of the student network.

[0172] In this step, for example, gradient backpropagation is performed only on the student network based on the total loss function to update the model weights of the student network.

[0173] S520: Apply an exponential moving average to the updated model weights of the student network, and use the processed weights as the model weights of the teacher network.

[0174] Specifically, in this step, the model weights of the teacher network are updated every K training iterations. The update method is as follows: first, the model weights of the student network are processed using an exponential moving average, and then the processed weights are copied to the teacher network model. Since the model weights of both the student network and the teacher network are updated throughout the training phase, and considering that the teacher network is actually an information extractor (tokenizer), "online information extraction" can be achieved. The training data of the information extractor (teacher network) and the student network are exactly the same, thus solving the problem of information differences between domains.

[0175] The following will further illustrate the specific steps of a self-supervised algorithm training method based on content awareness and self-distillation structure in conjunction with the figures.

[0176] For example, such as Figure 4 As shown, after step S500, the following steps are also included:

[0177] S600: Repeat steps S100 to S500 until a signal to stop training is received.

[0178] This disclosure presents a self-supervised algorithm training method based on content-aware and self-distillation structures. Specifically, it provides methods for obtaining energy maps, lossless and lossy representation sets, and updating model weights for student and teacher networks. It proposes a more efficient, memory-efficient, and effective training method that can better utilize unlabeled data to train models.

[0179] The following will refer to Figure 5 This disclosure describes a self-supervised algorithm training apparatus based on a content-aware and self-distillation structure, according to another embodiment of the present disclosure.

[0180] For example, such as Figure 5 As shown, a self-supervised algorithm training device 100 based on content-aware and self-distillation structure includes:

[0181] Image enhancement module 110 is used to process an image using a first enhancement method to obtain a first enhanced image, and to process an image using a second enhancement method to obtain a second enhanced image. The first enhancement method and the second enhancement method are different.

[0182] The energy map acquisition module 120 is used to acquire the energy map of the first enhancement map to obtain the first energy map, and to acquire the energy map of the second enhancement map to obtain the second energy map.

[0183] The characterization set acquisition module 130 is used to obtain a lossless characterization set and a lossy characterization set based on the first enhancement map, the second enhancement map, the first energy map, and the second energy map.

[0184] The self-distillation module 140 is used to input the lossy representation set into the student network in the self-distillation structure and the lossless representation set into the teacher network in the self-distillation structure, and obtain the total loss function based on the output results.

[0185] Update module 150 is used to update the model weights of the student network and the teacher network according to the total loss function.

[0186] This disclosure discloses a self-supervised algorithm training device based on content-aware and self-distillation structures, introducing two important core technologies: content-aware technology and self-distillation structures. Content-aware technology uses energy maps to quantify the importance of image content, expanding the reconstruction target from a single "reconstructed pixel" to "reconstructed energy," thereby improving the semantic level of the latent representations learned by the encoder and enhancing feature robustness. By introducing a self-distillation structure, online tokenizers are implemented, enabling one-step training and resolving the problem of information discrepancies between domains of the tokenizer, reducing training complexity and time consumption. Combining content-aware technology and self-distillation structures, a causal mechanism is introduced to constrain the learning process, thereby optimizing current contrastive self-supervised learning methods. This allows the training method to better utilize unlabeled data for model training. Models trained in this way can be adapted to other tasks in the image vision domain, including image classification, object detection, and instance segmentation, through targeted fine-tuning.

[0187] Furthermore, this embodiment also discloses an electronic device, including:

[0188] One or more processors;

[0189] A storage unit is used to store one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the self-supervised algorithm training method based on content awareness and self-distillation structure described above.

[0190] Furthermore, this embodiment also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, can implement the self-supervised algorithm training method based on content-aware and self-distillation structure described above.

[0191] The computer-readable medium may be included in the apparatus, device, or system disclosed herein, or it may exist independently.

[0192] In data processing, video stream data, image data, database data, and statistical results data are processed using data processing tools appropriate to their respective data types. For data management, integrated management tools are employed, enabling the creation of a unified data management catalog, index, and standards. The incoming data is processed and stored in stages, with full-cycle correlation and analysis.

[0193] The computer-readable storage medium may be any tangible medium that contains or stores a program, and may be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, optical fibers, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0194] The computer-readable storage medium may also include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code, specific examples of which include, but are not limited to, electromagnetic signals, optical signals, or any suitable combination thereof.

[0195] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0196] It is understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of this disclosure, and this disclosure is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this disclosure, and these modifications and improvements are also considered to be within the scope of protection of this disclosure.

Claims

1. A self-supervised algorithm training method based on content-aware and self-distillation structure, characterized in that, The method includes: The image is processed using a first enhancement method to obtain a first enhanced image, and the image is processed using a second enhancement method to obtain a second enhanced image. The first enhancement method and the second enhancement method are different. Obtain the energy map of the first enhancement map to obtain the first energy map; obtain the energy map of the second enhancement map to obtain the second energy map. Based on the first enhancement map, the second enhancement map, the first energy map, and the second energy map, a lossless characterization set and a lossy characterization set are obtained; The lossy representation set is input into the student network in the self-distillation structure, and the lossless representation set is input into the teacher network in the self-distillation structure. The total loss function is obtained based on the output results. The model weights of the student network and the model weights of the teacher network are updated according to the total loss function; The process of obtaining the lossless characterization set and the lossy characterization set based on the first enhancement map, the second enhancement map, the first energy map, and the second energy map includes: A data preprocessing result set is obtained based on the first enhanced map, the second enhanced map, the first energy map, and the second energy map; The data preprocessing result set is divided into equal-scale blocks to obtain the block-based data preprocessing result set; Perform a convolution operation on the preprocessed data set after segmentation to obtain a one-dimensional local embedding representation block set; A one-dimensional masked local embedding representation block set is obtained based on the one-dimensional local embedding representation block set and the learnable mask representation; A first set of classification placeholders is generated based on the one-dimensional mask local embedding representation block set, and a second set of classification placeholders is generated based on the one-dimensional local embedding representation block set. The lossy representation set is obtained based on the one-dimensional mask local embedding representation block set and the first classification placeholder set; the lossless representation set is obtained based on the one-dimensional local embedding representation block set and the second classification placeholder set. The process of inputting the lossy representation set into the student network of the self-distillation structure, inputting the lossless representation set into the teacher network of the self-distillation structure, and obtaining the total loss function based on the output results includes: The lossy representation set is input into the student network in the distillation structure to obtain the information category prediction probability and the first semantic category probability; The lossless representation set is input into the teacher network in the distillation structure to obtain the information category prediction probability and the second semantic category probability. The information loss value is obtained based on the predicted probability of the information category and the estimated probability of the information category. The semantic loss value is obtained based on the first semantic category probability and the second semantic category probability; The total loss function is obtained based on the information loss value and the semantic loss value.

2. The self-supervised algorithm training method based on content-aware and self-distillation structure according to claim 1, characterized in that, The step of obtaining the energy map of the first enhancement map to obtain the first energy map includes: Obtain the gradient of the first enhancement map to obtain the first energy map; The step of obtaining the energy map of the second enhancement map to obtain the second energy map includes: Obtain the gradient of the second enhancement map to obtain the second energy map.

3. The self-supervised algorithm training method based on content-aware and self-distillation structure according to claim 1, characterized in that, The step of inputting the lossy representation set into the student network in the distillation structure to obtain the information category prediction probability and the first semantic category probability includes: The one-dimensional local embedding representation block set and the first classification placeholder set in the lossy representation set are input into the encoder to obtain the student network output result; The student network output and the learnable mask representation in the lossy representation set are input into the predictor to obtain the information category prediction probability and the first semantic category probability.

4. The self-supervised algorithm training method based on content-aware and self-distillation structure according to claim 1, characterized in that, The step of inputting the lossless representation set into the teacher network in the distillation structure to obtain the information category prediction probability and the second semantic category probability includes: The one-dimensional local embedding representation block set and the second classification placeholder set in the lossless representation set are input into the encoder to obtain the teacher network output result; The output of the teacher network is input into the predictor to obtain the estimated probability of the information category and the probability of the second semantic category.

5. The self-supervised algorithm training method based on content-aware and self-distillation structure according to claim 1, characterized in that, The step of obtaining the total loss function based on the information loss value and the semantic loss value includes: The total loss function is obtained by summing the information loss value and the semantic loss value.

6. The self-supervised algorithm training method based on content-aware and self-distillation structure according to claim 1, characterized in that, The step of updating the model weights of the student network and the teacher network according to the total loss function includes: The student network is backpropagated using the total loss function to update its model weights.

7. The self-supervised algorithm training method based on content-aware and self-distillation structure according to claim 6, characterized in that, The step of updating the model weights of the student network and the teacher network according to the total loss function includes: The updated model weights of the student network are processed using an exponential moving average, and the processed weights are used as the model weights of the teacher network.

8. A self-supervised algorithm training device based on content-aware and self-distillation structure, characterized in that, The device includes: An image enhancement module is used to process an image using a first enhancement method to obtain a first enhanced image, and to process the image using a second enhancement method to obtain a second enhanced image, wherein the first enhancement method and the second enhancement method are different; An energy map acquisition module is used to acquire the energy map of the first enhancement map to obtain a first energy map, and to acquire the energy map of the second enhancement map to obtain a second energy map; The characterization set acquisition module is used to obtain a lossless characterization set and a lossy characterization set based on the first enhancement map, the second enhancement map, the first energy map, and the second energy map. The self-distillation module is used to input the lossy characterization set into the student network of the self-distillation structure, and the lossless characterization set into the teacher network of the self-distillation structure, and obtain the total loss function based on the output results; An update module is used to update the model weights of the student network and the model weights of the teacher network according to the total loss function; The process of obtaining the lossless characterization set and the lossy characterization set based on the first enhancement map, the second enhancement map, the first energy map, and the second energy map includes: A data preprocessing result set is obtained based on the first enhanced map, the second enhanced map, the first energy map, and the second energy map; The data preprocessing result set is divided into equal-scale blocks to obtain the block-based data preprocessing result set; Perform a convolution operation on the preprocessed data set after segmentation to obtain a one-dimensional local embedding representation block set; A one-dimensional masked local embedding representation block set is obtained based on the one-dimensional local embedding representation block set and the learnable mask representation; A first set of classification placeholders is generated based on the one-dimensional mask local embedding representation block set, and a second set of classification placeholders is generated based on the one-dimensional local embedding representation block set. The lossy representation set is obtained based on the one-dimensional mask local embedding representation block set and the first classification placeholder set; the lossless representation set is obtained based on the one-dimensional local embedding representation block set and the second classification placeholder set. The process of inputting the lossy representation set into the student network of the self-distillation structure, inputting the lossless representation set into the teacher network of the self-distillation structure, and obtaining the total loss function based on the output results includes: The lossy representation set is input into the student network in the distillation structure to obtain the information category prediction probability and the first semantic category probability; The lossless representation set is input into the teacher network in the distillation structure to obtain the information category prediction probability and the second semantic category probability. The information loss value is obtained based on the predicted probability of the information category and the estimated probability of the information category. The semantic loss value is obtained based on the first semantic category probability and the second semantic category probability; The total loss function is obtained based on the information loss value and the semantic loss value.

Images