Image processing method and device based on supervised contrastive learning

Abstract

The application provides an image processing method and device based on supervised contrast learning, wherein the method comprises: acquiring a to-be-processed image; inputting the to-be-processed image into a pre-trained visual self-attention model to obtain an image processing result; wherein the visual self-attention model is obtained by training and optimizing a global loss function, the global loss function comprises a diversity loss function, a contrast learning loss function and a cross-entropy loss function, and the process of training and optimizing does not use a data enhancement algorithm. The method trains and optimizes the visual self-attention model through a contrast learning method of learnable data enhancement, the contrast learning process does not need to use any data enhancement algorithm, thereby saving the expensive cost of designing data enhancement, and at the same time, by introducing the contrast learning loss function and the cross-entropy loss function, the quality of the contrast learning is ensured, thereby realizing the diversification of the visual self-attention model coding and improving the inference precision of the to-be-processed image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image processing method and apparatus based on supervised contrastive learning. Background Technology

[0002] Currently, neural network models have been shown to perform better as they become deeper and more complex.

[0003] In Natural Language Processing (NLP), the more data samples provided, the more complex the representations the model can learn, and the more beneficial they are for downstream tasks. However, in practice, high-quality labeled data is very expensive. Due to this limitation, unsupervised learning, which does not require data labeling, has achieved great success in NLP.

[0004] After being applied to computer vision, the Transformer model has proven to have state-of-the-art performance in many downstream tasks. Unlike word vector inputs in NLP, which are independent of each other, image inputs in computer vision are treated as a continuous space. Therefore, various image data augmentation methods have been developed in computer vision to increase data diversity, thereby significantly improving model performance.

[0005] When applied to contrastive learning, a common approach is to apply comparative and / or clustering methods to the encoding results of images using different enhancement methods, and minimize the contrastive loss to increase the diversity of negative samples or keep the positive samples consistent.

[0006] However, the quality of contrastive learning also depends on whether the data augmentation method used can generate images with different semantics. Manually designing data augmentation is expensive and may introduce bias.

[0007] Therefore, how to solve the problems of high cost and low quality of contrastive learning, which require the application of data augmentation algorithms in existing technologies, is an important issue that urgently needs to be addressed in the field of image processing technology. Summary of the Invention

[0008] This invention provides an image processing method and apparatus based on supervised contrastive learning, which overcomes the shortcomings of existing technologies where contrastive learning requires data augmentation algorithms, is costly, and has low quality, thereby achieving diversification of visual self-attention model encoding.

[0009] On one hand, the present invention provides an image processing method based on supervised contrastive learning, comprising: acquiring an image to be processed; inputting the image to be processed into a pre-trained visual self-attention model to obtain an image processing result; wherein the visual self-attention model is obtained by training and optimization through an overall loss function, the overall loss function including a diversity loss function, a contrastive learning loss function and a cross-entropy loss function, and the training and optimization process does not use data augmentation algorithms.

[0010] Further, the training and optimization of the visual self-attention model specifically includes: obtaining a visual self-attention model to be trained and optimized based on a given pre-trained model; wherein the pre-trained model is used as a teacher model, and the visual self-attention model to be trained and optimized is used as a student model; the comparative learning process of the student model is guided based on the intermediate results output by all teacher attention modules in the teacher model, and the cross-entropy loss function is introduced to supervise the comparative learning process; and the diversity loss function is used to diversify the parameters of the student attention modules in the student model, and the student model is trained to convergence; wherein the student model trained to convergence is the pre-trained visual self-attention model.

[0011] Further, the process of guiding the contrastive learning of the student model based on the intermediate results output by all teacher attention modules in the teacher model includes: inputting a first image into the teacher model and the student model respectively to obtain the teacher model processing result and the student model processing result of the first image; inputting a second image into the teacher model and the student model respectively to obtain the teacher model processing result and the student model processing result of the second image; and training the student model using the contrastive learning loss function based on the teacher model processing result and the student model processing result of the first image, and the teacher model processing result and the student model processing result of the second image; wherein the teacher model processing result is obtained based on the intermediate results output by all teacher attention modules in the teacher model, and the student model processing result is the output result of the last layer student self-attention module in the student model.

[0012] Further, the steps for obtaining the teacher model processing results are as follows: input the first image / second image into the teacher model to obtain the intermediate results output by all teacher attention modules in the teacher model; perform average pooling and uniform sampling operations on the intermediate results output by each teacher attention module to obtain the teacher model processing results of the first image / second image.

[0013] Furthermore, the student attention module in the student model includes an initial attention module for extracting image features, an intermediate attention module for image feature data augmentation, and a terminal attention module for capturing deep semantics of the image. Accordingly, the parameters of the student attention module in the student model are diversified by using the diversity loss function, including: using the diversity loss function to train and optimize the intermediate attention module in the student model so that the intermediate attention module model learns different data augmentations.

[0014] Furthermore, the contrastive learning loss function is expressed as follows:

[0015]

[0016] Where b represents the training batch, i represents the first image, and j represents the second image. s z i Image sampling results for the student model. t z k denoted as the image sampling result of the teacher model, f is the projection head with a classification layer, g is the cosine similarity, and τ is the hyperparameter of the model.

[0017] Furthermore, the diversity loss function is expressed as follows:

[0018] L D =-|ViTt[25%:75%]-ViTs[25%:75%]| 2

[0019] Where ViTt represents the teacher model and ViTs represents the student model.

[0020] Secondly, the present invention also provides an image processing apparatus based on supervised contrastive learning, comprising: an image acquisition module for acquiring an image to be processed; and an image processing module for inputting the image to be processed into a pre-trained visual self-attention model to obtain an image processing result; wherein the visual self-attention model is obtained by training and optimization through an overall loss function, the overall loss function including a diversity loss function, a contrastive learning loss function, and a cross-entropy loss function, and the training and optimization process does not use data augmentation algorithms.

[0021] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image processing method based on supervised contrastive learning as described above.

[0022] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method based on supervised contrastive learning as described above.

[0023] This invention provides an image processing method based on supervised contrastive learning. The method acquires the image to be processed and inputs it into a pre-trained visual self-attention model to obtain the output image processing result. The visual self-attention model is trained and optimized using an overall loss function, which includes a diversity loss function, a contrastive learning loss function, and a cross-entropy loss function. The training and optimization process does not use data augmentation algorithms. This method trains and optimizes the visual self-attention model using a contrastive learning method with learnable data augmentation. The contrastive learning process does not require any data augmentation algorithms, saving the expensive cost of designing data augmentation. Simultaneously, by introducing the contrastive learning loss function and the cross-entropy loss function, the quality of contrastive learning is ensured, thereby achieving diversity in the encoding of the visual self-attention model and improving the inference accuracy of the image to be processed. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating the supervised contrastive learning image processing method provided by the present invention.

[0026] Figure 2 This is a schematic diagram of the contrast fusion framework in the process of training a visual self-attention model provided by the present invention;

[0027] Figure 3 This is a schematic diagram illustrating the training and optimization of the visual self-attention model provided by the present invention;

[0028] Figure 4 This is a comparison diagram of the feature distribution between the supervised contrastive learning method provided by this invention and existing contrastive learning frameworks.

[0029] Figure 5 This is a schematic diagram of the image processing device based on supervised contrastive learning provided by the present invention;

[0030] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0032] It should be noted that existing contrastive learning methods require the use of different data augmentation methods to increase the diversity of data. This means that the quality of contrastive learning will depend on whether the data augmentation method used can generate images with different semantics. At the same time, manually designing data augmentation is costly and may introduce bias, thus affecting the effect of contrastive learning.

[0033] In view of this, the present invention proposes a supervised contrastive learning method. The network model learned by this method can perform higher-precision reasoning on images. Furthermore, since the contrastive learning process eliminates the need for manual data augmentation, it also reduces the cost requirements to a certain extent.

[0034] Specifically, Figure 1 A flowchart illustrating the image processing method based on supervised contrastive learning provided by this invention is shown.

[0035] like Figure 1 As shown, the method includes:

[0036] S110, Obtain the image to be processed;

[0037] S120, the image to be processed is input into a pre-trained visual self-attention model to obtain the image processing result; wherein, the visual self-attention model is obtained by training and optimization through an overall loss function, which includes a diversity loss function, a contrastive learning loss function, and a cross-entropy loss function, and the training and optimization process does not use data augmentation algorithms.

[0038] Understandably, in practical applications, the acquired image to be processed can be directly input into a pre-trained visual self-attention model. Thus, after processing by the visual self-attention model, the corresponding output result, i.e., the image processing result of the image to be processed, can be obtained.

[0039] It is worth mentioning that the visual self-attention model is trained and optimized using the supervised contrastive learning method provided in this embodiment. Specifically, it can be obtained through contrastive learning using an overall loss function, which consists of three parts: a diversity loss function, a contrastive learning loss function, and a cross-entropy loss function.

[0040] Specifically, the overall loss function can be expressed as follows:

[0041] L = L CE (predicts,labels)+αL D +βL CL

[0042] Where L is the overall loss function, L CE Let L be the cross-entropy loss function. D For the diversity loss function, L CL For the contrastive learning loss function, predicts represents the prediction results during training, labels represents the true labels, and α and β are the coefficients of the diversity loss function and the contrastive learning loss function, respectively. The parameters can be adjusted according to actual needs.

[0043] It should be noted that the network structure of the visual self-attention model is the same as that of the ViT (Vision Transformer) model, which will not be elaborated on here.

[0044] The diversity loss function is used to train the intermediate self-attention module of the visual self-attention model, so that the intermediate self-attention module of the visual self-attention model can learn different data augmentations, thereby extracting different image semantics of the same image, replacing the data augmentation algorithm used in the traditional contrastive learning process.

[0045] The contrastive learning loss function guides the contrastive loss by fusing the intermediate outputs of all attention modules (in the teacher model below), which pushes the final representation of negative samples away and pulls the final representation of positive samples closer together in the representation space.

[0046] The cross-entropy loss function is used to supervise the training process to ensure that the visual self-attention model can provide useful feature representations for downstream tasks.

[0047] The image processing result can be the classification result of the image to be processed, or the feature representation extracted from the image to be processed; no specific limitation is made here.

[0048] When the image processing result is a feature representation extracted from the image to be processed, the image processing result can be applied to a variety of different downstream tasks.

[0049] It should also be noted that the training process of the visual self-attention model, as well as the various loss functions applicable to the training process, will be described in detail in the following embodiments.

[0050] In this embodiment, an image to be processed is acquired and input into a pre-trained visual self-attention model to obtain the output image processing result. The visual self-attention model is trained and optimized using an overall loss function, which includes a diversity loss function, a contrastive learning loss function, and a cross-entropy loss function. The training and optimization process does not use data augmentation algorithms. This method trains and optimizes the visual self-attention model using a contrastive learning method with learnable data augmentation. The contrastive learning process does not require any data augmentation algorithms, saving the expensive cost of designing data augmentation. Simultaneously, by introducing the contrastive learning loss function and the cross-entropy loss function, the quality of contrastive learning is ensured, thereby achieving diversity in the encoding of the visual self-attention model and improving the inference accuracy of the image to be processed.

[0051] Based on the above embodiments, the visual self-attention model is further trained and optimized, specifically including: obtaining a visual self-attention model to be trained and optimized based on a given pre-trained model; wherein the pre-trained model is used as the teacher model and the visual self-attention model to be trained and optimized is used as the student model; the comparative learning process of the student model is guided based on the intermediate results output by all teacher attention modules in the teacher model, and the cross-entropy loss function is introduced to supervise the comparative learning process; and a diversity loss function is used to diversify the parameters of the student attention modules in the student model, and the student model is trained to convergence; wherein the student model trained to convergence is the pre-trained visual self-attention model.

[0052] Understandably, based on a given pre-trained ViT model, by replicating the weights of that pre-trained ViT model, another completely identical and independent pre-trained ViT model can be obtained, namely the visual self-attention model to be trained and optimized.

[0053] Thus, there are two independent pre-trained ViT models. One of these pre-trained ViT models can be used as the teacher model ViT. t Another pre-trained ViT model was used as the student model ViT. s .

[0054] It should be noted that during the training process, the weight parameters of the ViT model, which serves as the teacher model, do not participate in gradient descent. The weight parameters are fixed, and their role is to guide the training of another pre-trained ViT model (the student model).

[0055] Since the weights of the teacher model come from the pre-trained ViT model, and its gradient has converged to at least one local optimum, the weights of the student model and the teacher model can be diversified, preventing the student model and the teacher model from staying at the same optimum and ensuring that the teacher model and the student model can learn different representations at the same time.

[0056] The main goal of this embodiment for training and optimizing the visual self-attention model is to train a student model. The trained student model is the visual self-attention model used in actual applications and will be used for inference on the image to be processed.

[0057] Before describing the training and optimization process of the visual self-attention model in detail, a parameter symbol will be briefly described here.

[0058] For a ViT architecture, the model input fragment x has N sequences, and each sequence has d dimensions, i.e., x∈R. N×d .

[0059] The output of the m-th self-attention module is represented as z. m =[z m,0 ;z m,1 ;...;z m,N-1 ]∈R N×d , where z m,n ∈R 1 ×d , n∈[0,N) represents the nth single output sequence vector of the self-attention module.

[0060] Furthermore, in this embodiment, the set of outputs of all self-attention modules is defined as Z∈R. M×N×d .

[0061] For the teacher model and the student model, this embodiment uses a lower left subscript to distinguish the output of the teacher model and the student model.

[0062] For example, for the teacher model ViT t The following are examples:

[0063]

[0064] For the student model ViT s The following are examples:

[0065]

[0066] Therefore, for the teacher model, this embodiment will obtain all outputs of each self-attention module (teacher attention module). t z0; t z1;...; t z M-1 For the student model, this embodiment will obtain the output of the last self-attention module (student attention module). s z M-1,0 And it will be used as the encoding result.

[0067] In one specific embodiment, the visual self-attention model is trained and optimized. Specifically, based on the intermediate results output by all teacher attention modules in the teacher model, the contrastive learning process of the student model is guided by a contrastive learning loss function. In this way, high-level semantics from different attention modules can be combined to eliminate estimation bias.

[0068] Understandably, the comparative learning process of the student model is guided by the intermediate results output by all teacher attention modules in the teacher model. Specifically, firstly, it is necessary to obtain the processing results of the teacher model on the first and second images, i.e., the teacher model processing results, and secondly, to obtain the processing results of the student model on the first and second images, i.e., the student model processing results.

[0069] Figure 2 This diagram illustrates the contrast fusion framework used in training a visual self-attention model, as provided by the present invention.

[0070] like Figure 2 As shown, firstly, the processing results of the teacher model on the first image and the second image are obtained. Specifically, the first image / second image is input into the teacher model (including multiple self-attention modules) to obtain the intermediate results output by each teacher attention module in the teacher model. Average pooling and uniform sampling operations are performed on the intermediate results output by each teacher attention module to obtain the teacher model processing results of the first image / second image.

[0071] Expanding on this, for the intermediate results output by each teacher attention module in the teacher model... t z m ∈R N×d An average pooling layer can be used to perform average pooling operations on it, and return an average pooled value in R. 1×d Intermediate results in the process.

[0072] Then, in order to merge the hidden representations of all teacher attention modules and derive diverse high-level semantics... t z∈R 1 ×d A uniform sampler (Averageσ) can be used to sample along each teacher attention module, thereby diverting intermediate results from R... M ×d Mapping to R 1×d .

[0073] Sampling is performed using the function σ:

[0074] t z=σ( t Z}

[0075] in, tz represents the sampling result. t Z represents the output of the teacher attention module after average pooling.

[0076] Then, the sampling results t The z-value is input into the projection head to obtain the corresponding teacher model processing result f(. t z), including f( t z i ) and f( t z j ).

[0077] To obtain the processing results of the student model on the first and second images, specifically, the first / second image is input into the student model (which includes multiple self-attention modules), and the result output by the last student attention module in the student model is obtained. s z M,0 And use it as the encoding result of the first image / second image. s z.

[0078] Then, the encoded result is input into the projection head to obtain the corresponding student model processing result f( s z), including f( s z i ) and f( s z j ).

[0079] according to Figure 2 It can also be seen that only the intermediate attention module of the student model is trainable, while other modules are frozen and do not participate in the gradient descent process.

[0080] Based on the processing results of the teacher model and student model for the first image, and the processing results of the teacher model and student model for the second image, the student model can be trained using a contrastive learning loss function.

[0081] Specifically, the contrastive learning loss function is expressed as follows:

[0082]

[0083] Where b represents the training batch, i represents the first image, and j represents the second image. s z i The image sampling result of image i for the student model. t z k Let f be the image sampling result of the teacher model for image k, f be the projection head with classification layer, g be the cosine similarity, and τ be the hyperparameter of the model.

[0084] As can be seen from the contrastive learning loss function, since the teacher model does not participate in the gradient descent process, the contrastive learning loss function ignores the teacher model's processing results for two different teacher models on two different images. t z i and t z j The contrastive learning loss between the two student models is considered; meanwhile, the results of the student models processing two different images are taken into account. s z i and s z j No guidance information was included. s z i and s z j The contrastive learning loss between models is also meaningless and can be ignored. Therefore, the contrastive learning loss function in this embodiment only considers the contrastive learning loss between the processing results of different models on the same image or different images, that is... s z j and t z j between, s z i and t z i between, s z i and t z j Between and s z j and t z i The learning loss between comparisons.

[0085] By training and optimizing the student model using the contrastive learning loss function described above, the student model can push the final representation of negative samples further apart and bring the final representation of positive samples closer together in the representation space.

[0086] In another specific embodiment, the visual self-attention model is trained and optimized. Specifically, the student attention module in the student model includes an initial attention module for extracting image features, an intermediate attention module for image feature data augmentation, and a terminal attention module for capturing image depth semantics. Accordingly, a diversity loss function is used to diversify the parameters of the student attention module in the student model, including: using a diversity loss function to train and optimize the intermediate attention module in the student model so that the intermediate attention module model learns different data augmentations.

[0087] Understandably, the core idea of ​​this embodiment is to utilize student attention modules as an automatic data augmentation mechanism. In the student model composed of M student attention modules, some initial blocks (initial attention modules) act as feature extractors, extracting features from the input image; the last few blocks (terminal attention modules) are trained to capture the deep semantics of the image and generate encodings; the intermediate blocks (intermediate attention modules) play a crucial role in processing the features extracted by the initial attention modules and passing them to the terminal attention modules. Essentially, these intermediate attention modules exist as data augmentation for image features.

[0088] Based on the aforementioned principles, this embodiment trains the intermediate attention module to encourage diversity in the feature processing process. Since the intermediate attention module is a self-attention block, diverse weights are sufficient to promote diversity in processing results. Therefore, this embodiment selects to train 25% to 75% of the intermediate self-attention modules to learn different data augmentations.

[0089] Specifically, in this embodiment, the self-attention modules for the middle 25% to 75% are trained by introducing a diversity loss function, which is expressed as follows:

[0090] L D =-|ViTt[25%:75%]-ViTs[25%:75%]| 2

[0091] Where ViTt represents the teacher model and ViTs represents the student model.

[0092] It should be noted that the intermediate attention module selects 25% to 75% of the self-attention module. The range of "25% to 75%" is not completely fixed and can be adjusted according to actual needs.

[0093] Based on the aforementioned diversity loss function, a penalty can be added to the weight similarity between the student model ViTs and the teacher model ViTt to promote training with different data augmentations, thereby enabling the student model to generate different representations for the same image.

[0094] In yet another specific embodiment, a cross-entropy loss function is also introduced during the training and optimization of the visual self-attention model.

[0095] Understandably, while the aforementioned diversity loss function can promote training with different data augmentations, it may cause the student model to deviate too much from the teacher model, preventing the student model from learning useful data augmentations.

[0096] To address this issue, this embodiment introduces label information to supervise the training process of the student model, ensuring that the student model ultimately learns an optimal encoding representation for downstream tasks. Specifically, this embodiment adds a cross-entropy loss function to train a classification head on the student model and guides gradient propagation in the most suitable direction, achieving a 10-fold speed improvement compared to traditional unsupervised frameworks.

[0097] It should be noted that the classification head in this embodiment is independent of the projection head used in the contrast loss function described above.

[0098] In another specific embodiment, the process of training a student model using a general contrastive loss function is given.

[0099] Specifically, assuming the training batch size is b, data augmentation increases the batch size to 2b.

[0100] For each individual training step, there are two different inputs x. i and x j Furthermore, a general contrastive loss function for visualization representation is proposed, which can be calculated as:

[0101]

[0102] Where s i,j It is the similarity between [CLS] tokens, and τ is a hyperparameter used to make the projection more uniform.

[0103] Student model ViT s The output token should be provided by the teacher model ViT t The fusion results provide guidance. For the i-th input, given a batch of data of size b, there is a set of outputs:

[0104] P = { s z 1 , s z 2 ,..., s z b}∪{ t z 1 , t z 2 ,…, t z b}

[0105] For each p i For any P, the general contrastive loss function becomes:

[0106]

[0107] Where f is a projection head with an MLP (Multilayer Perceptron) layer, and g(u,v) = u T ·v / (|u|2|v|2) is the cosine similarity.

[0108] Therefore, the overall general comparison loss is:

[0109]

[0110] The above describes the specific process of training a student model using general contrastive loss.

[0111] In this embodiment, the contrastive learning loss function pushes the final representations of negative samples further apart and brings the final representations of positive samples closer together in the representation space. The diversity loss function trains the student model to generate different representations for the same image. The cross-entropy loss function ensures that the student model can generate representations meaningful for downstream tasks after training. This method trains and optimizes the visual self-attention model using a learnable data-augmented contrastive learning approach. The contrastive learning process does not require any data augmentation algorithms, saving the expensive cost of designing data augmentation. Simultaneously, by introducing the contrastive learning loss function and the cross-entropy loss function, the quality of contrastive learning is guaranteed, thereby achieving diversification of the visual self-attention model's encoding and improving the inference accuracy of the images being processed.

[0112] In some embodiments, Figure 3 A schematic diagram illustrating the training optimization of the visual self-attention model provided by this invention is shown.

[0113] like Figure 3 As shown, the training process of the visual self-attention model (student model) involves three loss functions: Diverse Loss, Cross-Entropy Loss (CE loss), and Contrastive Learning Loss (NT-Xent Loss).

[0114] A diversity loss function is used in the training process of the intermediate attention modules of the student model so that the intermediate 25% to 75% of the self-attention modules can learn different data augmentations.

[0115] The cross-entropy loss function is used to train the classification head on the student model and guide gradient propagation in the most suitable way, thereby achieving a 10x speedup compared to traditional unsupervised frameworks.

[0116] Contrastive learning loss is used to enable the student model to push the final representation of negative samples away and pull the final representation of positive samples closer in the representation space.

[0117] exist Figure 3 On the left, the double arrows between the student and teacher models represent the repulsive relationship in contrastive learning loss; in Figure 3 The right side shows the contrastive learning loss section, with double arrows indicating the attractive relationship in the contrastive learning loss.

[0118] In some embodiments, in order to verify whether the supervised contrastive learning method provided by the present invention diversifies the representation encoded by the visual self-attention model, the feature distribution of the final representation is plotted.

[0119] Specifically, Figure 4 The diagram shows a comparison of the feature distributions of the supervised contrastive learning method provided by this invention and existing contrastive learning frameworks.

[0120] like Figure 4 As shown, from top to bottom, the feature distributions include the feature distributions under Supervised Predictive Learning (Pretrained), Unsupervised Contrastive Learning (MoCo V3), and Supervised Contrastive Learning (CoFu) provided by this invention.

[0121] Figure 4 This represents the final representations obtained by various contrastive learning methods on the CIFAR-100 dataset. (The representations are then presented in R...) 2 From the perspective of the Gaussian kernel density estimate (KDE) and the von Mises-Fischer (vMF) KDE, the uniformity of the characteristic distribution (generated by the projector head) is analyzed.

[0122] The leftmost graph represents the feature distribution of all categories; the two rightmost graphs visualize the feature distribution of selected categories (class 0 and class 3). The supervised contrastive learning method provided by this invention exhibits the most uniform distribution in the vector space.

[0123] In other words, according to Figure 4 As can be seen, compared with other existing contrastive learning frameworks, the supervised contrastive learning method provided by this invention (the training optimization process of the visual self-attention model in this invention) successfully increases the diversity of representations.

[0124] To demonstrate the accuracy of the comparison results, this embodiment verifies the supervised contrastive learning method provided by the present invention using ImageNet-1k, CIFAR-10 / 100, OxfordFlowers-102, and Oxford-IIIT-Pet benchmarks. The results show that the present invention significantly improves the performance of the pre-trained ViT model.

[0125] Furthermore, this embodiment also compares the supervised contrastive learning method provided by the present invention with five commonly used data augmentation algorithms mixup, cutout, cutmix, gridmask and augmi, as well as four contrastive learning frameworks mocov3, simclr, byol and swav. The results all show that the supervised contrastive learning method provided by the present invention has a better contrastive learning effect.

[0126] Based on the above, the image processing method based on supervised contrastive learning provided by this invention has the following advantages:

[0127] 1) This invention proposes a supervised contrastive learning method based on a self-attention mechanism of multi-view fusion, which has achieved state-of-the-art performance on downstream datasets;

[0128] 2) This invention is the first framework that does not require manual data augmentation, achieved through our proposed novel loss function, which diversifies the feature distribution of the representation, and is validated by uniformity analysis on a hypersphere plot;

[0129] 3) This invention achieves a 10-fold speed improvement compared to traditional unsupervised frameworks by introducing cross-entropy loss and guiding gradient propagation in the most suitable direction.

[0130] It should also be noted that the supervised contrastive learning method described in this invention can be applied not only to image classification tasks, but also to other downstream tasks, such as multi-object tracking tasks and image segmentation tasks.

[0131] Figure 5 A schematic diagram of the image processing apparatus based on supervised contrastive learning provided by the present invention is shown.

[0132] like Figure 5 As shown, the device includes: an image acquisition module 510 for acquiring an image to be processed; and an image processing module 520 for inputting the image to be processed into a pre-trained visual self-attention model to obtain an image processing result; wherein the visual self-attention model is obtained by training and optimization through an overall loss function, which includes a diversity loss function, a contrastive learning loss function, and a cross-entropy loss function, and the training and optimization process does not use data augmentation algorithms.

[0133] In this embodiment, the image to be processed is acquired by the image acquisition module 510, and the image processing module 520 inputs the image to be processed into a pre-trained visual self-attention model to obtain the output image processing result. The visual self-attention model is trained and optimized using an overall loss function, which includes a diversity loss function, a contrastive learning loss function, and a cross-entropy loss function. The training and optimization process does not use data augmentation algorithms. This device trains and optimizes the visual self-attention model using a contrastive learning method with learnable data augmentation. The contrastive learning process does not require any data augmentation algorithms, saving the expensive cost of designing data augmentation. Simultaneously, by introducing the contrastive learning loss function and the cross-entropy loss function, the quality of contrastive learning is ensured, thereby achieving diversity in the encoding of the visual self-attention model and improving the inference accuracy of the image to be processed.

[0134] It should be noted that the image processing device based on supervised contrastive learning provided in this embodiment can be referred to in correspondence with the image processing method based on supervised contrastive learning described above, and will not be repeated here.

[0135] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a supervised contrastive learning-based image processing method. This method includes: acquiring an image to be processed; inputting the image to be processed into a pre-trained visual self-attention model to obtain an image processing result; wherein the visual self-attention model is trained and optimized using an overall loss function, which includes a diversity loss function, a contrastive learning loss function, and a cross-entropy loss function. The training and optimization process does not use data augmentation algorithms.

[0136] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image processing method based on supervised contrastive learning provided by the above methods. The method includes: acquiring an image to be processed; inputting the image to be processed into a pre-trained visual self-attention model to obtain an image processing result; wherein the visual self-attention model is obtained by training and optimization through an overall loss function, the overall loss function including a diversity loss function, a contrastive learning loss function, and a cross-entropy loss function, and the training and optimization process does not use data augmentation algorithms.

[0138] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing method based on supervised contrastive learning, characterized in that, include: Obtain the image to be processed; The image to be processed is input into a pre-trained visual self-attention model to obtain the image processing result; The visual self-attention model is obtained through training and optimization using an overall loss function, which includes a diversity loss function, a contrastive learning loss function, and a cross-entropy loss function. The training and optimization process does not use data augmentation algorithms. The training and optimization of the visual self-attention model specifically includes: Based on a given pre-trained model, a visual self-attention model to be trained and optimized is obtained; wherein the pre-trained model is used as the teacher model and the visual self-attention model to be trained and optimized is used as the student model. Based on the intermediate results output by all teacher attention modules in the teacher model, the comparative learning process of the student model is guided, and the cross-entropy loss function is introduced to supervise the comparative learning process. In addition, the diversity loss function is used to diversify the parameters of the student attention module in the student model, and the student model is trained to convergence. Among them, the student model trained to convergence is the pre-trained visual self-attention model; The student attention module in the student model includes an initial attention module for extracting image features, an intermediate attention module for image feature data augmentation, and a terminal attention module for capturing deep semantics of the image. Accordingly, the diversity loss function is used to diversify the parameters of the student attention module in the student model, including: The intermediate attention module in the student model is trained and optimized using the diversity loss function, so that the intermediate attention module model can learn different data augmentations.

2. The image processing method based on supervised contrastive learning according to claim 1, characterized in that, The comparative learning process of the student model, guided by intermediate results output by all teacher attention modules in the teacher model, includes: The first image is input into the teacher model and the student model respectively to obtain the teacher model processing result and the student model processing result of the first image; The second image is input into the teacher model and the student model respectively to obtain the teacher model processing result and the student model processing result of the second image; Based on the teacher model processing results and student model processing results of the first image, and the teacher model processing results and student model processing results of the second image, the student model is trained using the contrastive learning loss function. The teacher model processing result is obtained based on the intermediate results output by all teacher attention modules in the teacher model, and the student model processing result is the output result of the last layer student self-attention module in the student model.

3. The image processing method based on supervised contrastive learning according to claim 2, characterized in that, The steps for obtaining the teacher model processing results are as follows: The first image / second image is input into the teacher model to obtain intermediate results output by all teacher attention modules in the teacher model; Average pooling and uniform sampling operations are performed on the intermediate results output by each of the teacher attention modules to obtain the teacher model processing results for the first image / second image.

4. The image processing method based on supervised contrastive learning according to any one of claims 1-3, characterized in that, The contrastive learning loss function is expressed as follows: Where b represents the training batch, i represents the first image, and j represents the second image. s z i Image sampling results for the student model. t z k denoted as the image sampling result of the teacher model, f is the projection head with a classification layer, g is the cosine similarity, and τ is the hyperparameter of the model.

5. The image processing method based on supervised contrastive learning according to any one of claims 1-3, characterized in that, The diversity loss function is expressed as follows: L D =-|White[25%:75%]-White[25%:75%]| 2 Where ViTt represents the teacher model and ViTs represents the student model.

6. An image processing apparatus based on supervised contrastive learning, employing the image processing method based on supervised contrastive learning as described in any one of claims 1-5, characterized in that, include: The image acquisition module is used to acquire the image to be processed. The image processing module is used to input the image to be processed into a pre-trained visual self-attention model to obtain the image processing result; The visual self-attention model is obtained through training and optimization using an overall loss function, which includes a diversity loss function, a contrastive learning loss function, and a cross-entropy loss function. The training and optimization process does not use data augmentation algorithms.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image processing method based on supervised contrastive learning as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image processing method based on supervised contrastive learning as described in any one of claims 1 to 5.

Images