Self-supervised fine-grained feature learning method and system based on global and local information

By introducing global attention distillation module and local key area learning module in comparison learning, using global and local information to extract local features, the problem of insufficient feature learning in fine-grained visual recognition tasks in the existing technology is solved, and the fine-grained visual recognition task is significantly improved Performance of granular image recognition.

CN118799664BActive Publication Date: 2024-11-26OCEAN UNIV OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411273058.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-12
Publication Date
2024-11-26
Estimated Expiration
2044-09-12

AI Technical Summary

Technical Problem

Existing contrast learning methods do not perform well in fine-grained visual recognition tasks because they tend to learn coarse-grained features, ignoring the local features of subtle discrimination.

Method used

The global attention distillation module and the local key area learning module were designed. Through the learning of global and local information, the learning of local important areas is emphasized, and the global attention distillation loss and local discrimination loss are used to extract local fine features.

Benefits of technology

The performance of self-supervised fine-grained image recognition characterization learning is significantly improved, and the model's ability to capture subtle differences is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118799664B_ABST
    Figure CN118799664B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of image processing technology, and discloses a self-supervised fine-grained feature learning method and system based on global and local information. The method comprises the following steps: step 1, generating a global view; step 2, generating a global view sample pair; step 3, segmenting the global view and extracting global features; step 4, collecting global information; step 5, generating an attention map using the global information; step 6, marking local key areas and obtaining local features; step 7, loss calculation: the total loss includes global contrast loss, attention distillation loss and local discrimination loss; the present invention further emphasizes local important areas in the process of global contrast learning of images and finds subtle differences in these local areas suitable for fine-grained visual recognition tasks, and improves the ability to extract local fine features through learning global and local information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and relates to deep learning and fine-grained image recognition technology, and in particular to a self-supervised fine-grained feature learning method and system based on global and local information. Background Art

[0002] In the field of computer vision, feature learning is a key step in solving visual tasks such as image recognition and target detection. Compared with supervised learning methods, self-supervised learning methods can automatically learn generalized representations from a large amount of unlabeled data, and produce visual representations that compete with supervised learning in some downstream tasks, which has attracted widespread attention. Especially when faced with challenges such as difficult data labeling and high cost consumption, self-supervised learning methods that do not require labeled data have become an effective way to solve practical problems, such as self-supervised learning to assist in feature learning of large models. Contrastive learning has been the mainstream strategy for self-supervised learning in recent years, and has demonstrated advanced feature learning capabilities on many general datasets. However, unlike general object recognition tasks, fine-grained visual recognition tasks aim to distinguish similar subcategories of a given object category, requiring the ability to capture subtle differences between subcategories.

[0003] Current contrastive learning methods tend to learn coarse-grained features suitable for general object recognition tasks, but do not sufficiently learn features that are crucial for fine-grained visual recognition tasks and can reflect subtle distinctions. Specifically, current contrastive learning methods consider two views from the same image as positive pairs and two views from different images as negative pairs, and learn discriminative features by maximizing the similarity of positive pairs and the difference of negative pairs. This idea enables the model to learn global discriminative features of the overall visual similarity of views, but ignores many refined local features that are beneficial to fine-grained visual recognition tasks, resulting in the poor performance of current contrastive learning methods in solving fine-grained visual recognition problems.

[0004] In order to further enhance the ability of the self-supervised contrastive learning method to capture fine-grained features, the present invention makes full use of the global information of the image, mines the important local areas in the image, supplements the learning of local fine-grained features, and designs a global attention distillation module and a local key area learning module. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a self-supervised fine-grained feature learning method and system based on global and local information, which aims to further emphasize local important areas in the process of global contrast learning of images and discover subtle differences suitable for fine-grained visual recognition tasks in these local areas. By learning global and local information, the ability to extract local fine features is improved. In order to explicitly supervise the model's attention to local important areas, (1) a global attention distillation module is designed: each view is divided into multiple equal-sized but non-overlapping image blocks. Taking the image block as a unit, in the process of global contrast learning, different methods are used to integrate global information for different network branches, and the global information obtained by different methods is mapped to the same contrast space, represented in the form of attention map, and the attention distillation loss is calculated; under the constraint of attention distillation loss, the learning of important features in global information is emphasized; (2) a local key area learning module is designed: based on global information, the importance of each image block in the global view is evaluated, and through the position mask sampling method, relatively important image blocks, namely local key areas, are selectively retained at the feature level according to the mask ratio; under the constraint of local discriminant loss, the information of local key areas is strengthened, and fine-grained features are learned from local key areas.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] First, the present invention provides a self-supervised fine-grained feature learning method based on global and local information, comprising the following steps:

[0008] Step 1: Generate a global view:

[0009] Input a batch of images, for each original image I in the same batch e , two different but equal-sized global views are generated through two random enhancement methods and , each global view is denoted as x e ;

[0010] Step 2: Generate global view sample pairs:

[0011] Two different global views generated from the same image but with different enhancement methods are considered as positive sample pairs, and two global views generated from different images and with different enhancement methods are considered as negative sample pairs;

[0012] Step 3: Segment the global view and extract global features:

[0013] Each global view x obtained in step 1 e Divide into n equal-sized but non-overlapping image blocks V e, all image blocks are flattened into sequences and linearly mapped into a D-dimensional space using a Transformer-based encoder and by the encoder The momentum encoder generated by the momentum update of the parameters Global view and Encode and obtain global features , , and ;

[0014] Step 4: Collect global information:

[0015] Different methods are used to collect the global view and Global information and , each global view x e The global information is represented as G e ;

[0016] Step 5: Generate attention map using global information:

[0017] For global information and global information Different methods are used to generate attention maps and , each global view x e The attention map is represented as ;

[0018] Step 6: Mark local key areas and obtain local features:

[0019] According to the global information G e , sort the attention weights of all image blocks in each global view in reverse order; sample image blocks with high attention weights, i.e., local key areas, through the position mask sampling method, and record the positions of local key areas; generate a mask matrix, multiply the mask matrix by the global features element by element, and obtain local features that only retain the information of local key areas .

[0020] Furthermore, the self-supervised fine-grained feature learning method based on global and local information further includes step 7, loss calculation: the total loss includes global contrast loss, attention distillation loss and local discrimination loss;

[0021] Among them, global contrast loss: use the contrast loss function to calculate the global contrast loss of the global features extracted in step 3 ;

[0022] Attention distillation loss: The attention distillation loss is calculated using the Kullback–Leibler divergence loss function on the attention map generated in step 5. ;

[0023] Local discriminant loss: Use the cross entropy loss function to calculate the local discriminant loss for the local features obtained in step 6 ;

[0024] Total loss The calculation formula is:

[0025] ;

[0026] in, , , is the loss weight; during the optimization process, the AdamW optimizer is used to dynamically adjust the parameters of the model, but only the encoder Participate in the back propagation process of the network, and the momentum encoder Does not participate in the gradient back propagation process.

[0027] Furthermore, the steps of step 1 are as follows: during the training process, the batch size selected each time is E, and each original image I in the same batch e After random enhancement method ① and random enhancement method ②, two different but equal global views are generated. and global view ,The random enhancement method includes a combination of random cropping commanded by random parameters, Gaussian blur and horizontal folding.

[0028] Furthermore, in step 2, set It is the global view generated by the original image I1 through the enhancement method ①. It is the global view generated by the original image I1 through enhancement method ②. and are mutually positive sample pairs; set It is the global view generated by the original image I2 through enhancement method ②. and are mutually negative sample pairs; in the same batch, the view set U generated by enhancement method ① 1 Include Positive samples And negative sample set Similarly, the view set U generated by enhancement method ② 2 Include Positive samples And negative sample set .

[0029] Furthermore, in step 4, for the global view , records the encoder during the network back propagation process The output feature map and output result of the last Transformer layer of the , by calculating the gradient to reflect the attention weight of each image block in the global view to the output result, to obtain the global information of the entire view ;

[0030] For global view , momentum encoder Each Transformer layer relies on a multi-head self-attention mechanism to calculate the attention weights that reflect the relative relationship between each image block and other image blocks, collect and calculate the attention weights of all layers of the entire view, and obtain the global information of the entire view. .

[0031] Furthermore, step 4, collecting global information is as follows:

[0032] For the view generated by enhancement method ①, the encoder in the back propagation process of the global contrast network The feature map and decision results of the last Transformer layer are used to generate the class activation map, which reflects the important information in the view. Specifically, The output result y is back-propagated to calculate the gradient relative to the output feature map Y of the last Transformer layer , perform global average pooling on the gradient to obtain the weight vector W of each feature channel in the view, multiply the weight vector W by the output feature map Y, and obtain the attention weight G of the view 1 ;

[0033] For the view generated by enhancement method ②, the momentum encoder Each Transformer layer in the above example relies on the multi-head self-attention mechanism to focus on important information. Specifically, each Transformer layer will calculate the attention weights between the vector Q of the query image block and the vector K of other image blocks, and obtain the attention weight list A containing all layers, where each layer of the attention weight list contains the attention weights corresponding to n image blocks. ; Collect and calculate the attention weights of all layers of this view as G 2 .

[0034] Furthermore, in step 5, the specific steps of generating an attention map using global information are as follows:

[0035] For the view generated by enhancement method ①, the global information G 1 After normalization, we get ,right Upsample to generate an attention map H that is consistent with the size of the corresponding global view 1 ;

[0036] For the view generated by enhancement method ②, the global information G 2 Perform smoothing, binning, and upsampling to generate an attention map H that is consistent with the size of the corresponding global view 2 .

[0037] Furthermore, for step 6, the details are as follows:

[0038] Step 601, sorting in reverse order: the global information is composed of the attention weight value of each image block in the global view. These weight values ​​are arranged according to the position information of the corresponding image blocks to form a sequence, and are sorted in reverse order according to the size of the attention weight value;

[0039] Step 602, sampling: Calculate the sampling number according to the mask ratio, sample the attention weight values ​​in sequence until the sampling is completed, record the last sampled weight value and set it as the threshold;

[0040] Step 603, marking: in the global view, according to the position sequence of the image blocks, the threshold and the attention weight of each image block are compared in turn. If the threshold is greater than the attention weight of a certain image block, the image block at this position is recorded as a local key area, otherwise, it is recorded as a local non-key area; according to the position sequence of the image blocks, each image block is marked in turn. For the image block recorded as a local key area, the mask value of the position is marked as 1, otherwise, its mask value is marked as 0, and a mask matrix with the same dimension as the global feature is generated;

[0041] Step 604: Generate local features: multiply the mask matrix and the global features element by element to obtain local features that only retain information of the local key area. .

[0042] The present invention also provides a self-supervised fine-grained feature learning system based on global and local information, which is used to implement the self-supervised fine-grained feature learning method based on global and local information as described above, wherein the system includes a global attention distillation module and a local key area learning module.

[0043] The global attention distillation module divides each view into multiple equal-sized but non-overlapping image blocks. In the process of global contrastive learning, different methods are used for different network branches to integrate global information, and the global information obtained by different methods is mapped to the same contrast space, represented in the form of an attention map, and the attention distillation loss is calculated. Under the constraint of the attention distillation loss, the learning of important features in the global information is emphasized.

[0044] The local key area learning module: evaluates the importance of each image block in the global view according to the global information, and selectively samples the image blocks with high attention weights, i.e., the local key areas, at the feature level according to the mask ratio through the position mask sampling method; strengthens the information of the local key areas under the constraint of the local discriminant loss, and learns fine-grained features from the local key areas.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] (1) In the process of global image understanding, important local regions are further emphasized and subtle differences are discovered within these local regions.

[0047] A global attention distillation module is designed. In the process of global contrastive learning, different methods are used to integrate global information for different network branches. The global information obtained by different methods is mapped to the same contrast space and represented in the form of an attention map. The attention distillation loss is calculated. Under the constraint of the attention distillation loss, the learning of important features in the global information is emphasized.

[0048] A local key region learning module is designed to learn local discriminative features in important local regions. The local key region of any view and any image view are considered as sample pairs, and fine-grained features are learned from the local key region under the constraint of local contrast loss. Through global and local feature learning optimization, the model's ability to capture subtle differences is enhanced.

[0049] (2) The present invention is innovative in exploring fine-grained information and enhancing the applicability of self-supervised contrastive learning methods in fine-grained visual recognition problems, significantly improving the performance of self-supervised fine-grained image recognition representation learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0051] Figure 1 is a flow chart of the method of the present invention;

[0052] Figure 2 Schematic diagram of the structure of the global attention distillation module of the present invention;

[0053] Figure 3 It is a structural diagram of the local key area learning module of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0055] like Figure 1-Figure 3 As shown in FIG, the self-supervised fine-grained feature learning method based on global and local information includes the following steps:

[0056] Step 1: Generate a global view:

[0057] Input a batch of images, for each original image I in the same batch e , two different but equal-sized global views are generated through two random enhancement methods and , each global view is denoted as x e .

[0058] The specific steps of step 1 are as follows: During the training process, the batch size selected each time is E, and each original image in the same batch After random enhancement method ① and random enhancement method ②, two different but equal global views are generated. and global view , where wd represents the width of the original image, ht represents the height of the original image, and ch is the number of color channels. , each global view , where hw represents the size of the global view, and the random enhancement method includes a combination of random cropping directed by random parameters, Gaussian blur, and horizontal folding.

[0059] Step 2: Generate global view sample pairs:

[0060] Two different global views generated from the same image but through different enhancement methods are regarded as a positive sample pair, and two global views generated from different images and through different enhancement methods are regarded as a negative sample pair.

[0061] In step 2, set It is the global view generated by the original image I1 through the enhancement method ①. It is the global view generated by the original image I1 through enhancement method ②. and are mutually positive sample pairs; set It is the global view generated by the original image I2 through enhancement method ②. and are mutually negative sample pairs; in the same batch, the view set U generated by enhancement method ① 1 Include Positive samples And negative sample set Similarly, the view set U generated by enhancement method ② 2 Include Positive samples And negative sample set .

[0062] Step 3: Segment the global view and extract global features:

[0063] First, each global view obtained in step 1 Divide the image into n equal-sized but non-overlapping blocks , is the number of image blocks, hw represents the size of the global view, ch is the number of color channels, and ps represents the size of the image block; then, all image blocks are flattened into a sequence and linearly mapped into a D-dimensional space, using an encoder with the same encoder structure as Vision Transformer or a variant of Vision Transformer such as Swin Transformer and by the encoder The momentum encoder generated by the momentum update of the parameters Global view and Encode and obtain global features , , and That is, the global view of the same batch generates a global feature set , , , ,in, Include The positive sample set And negative sample set , similarly, Include The positive sample set And negative sample set .

[0064] It should be noted that after extracting the global features in step 3, the global feature set is used to calculate the global contrast loss , the specific formula can be found in the detailed description of step 7 below.

[0065] Step 4: Collect global information:

[0066] Different methods are used to collect the global view and Global information and , each global view x e The global information is represented as Ge .

[0067] For global view , records the encoder during the network back propagation process The output feature map and output result of the last Transformer layer of the , by calculating the gradient to reflect the attention weight of each image block in the global view to the output result, to obtain the global information of the entire view .

[0068] For global view , momentum encoder Each Transformer layer relies on a multi-head self-attention mechanism to calculate the attention weights that reflect the relative relationship between each image block and other image blocks, collect and calculate the attention weights of all layers of the entire view, and obtain the global information of the entire view. .

[0069] As a preferred implementation, the global information is collected as follows:

[0070] For the view generated by enhancement method ①, the encoder in the back propagation process of the global contrast network The feature map and decision results of the last Transformer layer are used to generate the class activation map, which reflects the important information in the view. Specifically, The output result y is back-propagated to calculate the gradient relative to the output feature map Y of the last Transformer layer , perform global average pooling on the gradient to obtain the weight vector W for each feature channel in the view,

[0071] ;

[0072] Multiply the weight vector W and the output feature map Y to get the attention weight of the view G 1 , .

[0073] For the view generated by enhancement method ②, the momentum encoder Each Transformer layer in relies on the multi-head self-attention mechanism to focus on important information. Specifically, each Transformer layer will calculate the attention weight between the vector Q of the query image block and the vector K of other image blocks, representing the connection between the image blocks in the current layer (the i-th layer):

[0074] ;

[0075] Get a list of attention weights for all layers , where each layer of attention weight list contains the attention weights corresponding to n image blocks The comprehensive connection between image patches in all layers reflects the global information of the entire view. The attention weights of all layers of the view are collected and calculated as ;in, Represents the dot product operation.

[0076] Step 5: Generate attention map using global information:

[0077] For global information and global information Different methods are used to generate attention maps and , each global view x e The attention map is represented as .

[0078] The specific steps of generating an attention map using global information are as follows:

[0079] For the view generated by enhancement method ①, the global information G 1 After normalization, we get ,right Upsample to generate an attention map H that is consistent with the size of the corresponding global view 1 .

[0080] For the view generated by enhancement method ②, the global information G 2 Perform smoothing, binning, and upsampling to generate an attention map H that is consistent with the size of the corresponding global view 2 .

[0081] As a specific implementation, the specific steps of generating an attention map using global information are as follows:

[0082] For the view generated by enhancement method ①, the global information G 1 Perform normalization:

[0083] ;

[0084] right Upsample to generate an attention map consistent with the corresponding global view size hw×hw .

[0085] For the view generated by enhancement method ②, the global information G 2 Smoothing, binning mapping and upsampling are performed. The smoothing method can be mean smoothing, median smoothing, Gaussian smoothing, etc. The binning method can be equal-width binning, equal-frequency binning, cluster binning, etc. Taking Gaussian smoothing and equal-width binning as examples, the details are as follows:

[0086] Step 501: Reconstruct global information: Reconstruct global information G 2 Refactored to Matrix of size , the element in the i-th row and j-th column of the matrix is ​​denoted by ,in, ;

[0087] Step 502, define the Gaussian kernel: The Gaussian kernel Ker is usually defined by the following formula:

[0088] ;

[0089] Among them, b represents the horizontal offset relative to the center of the core, and c represents the vertical offset relative to the center of the core. represents the standard deviation of the Gaussian kernel;

[0090] Step 503: Apply Gaussian smoothing: Each element of Substituting into the Gaussian smoothing formula to calculate the value after Gaussian smoothing, it can be expressed as :

[0091] ;

[0092] Where r represents the nuclear radius, usually taken as , so that the matrix Convert to Matrix ;

[0093] Step 504: perform binning: After flattening to a one-dimensional vector, record its original index and the smoothed value After sorting the elements from small to large, we get a new vector B, set the number of bins d, and get the width of each bin as , distribute the data to the corresponding boxes, record the number of elements in each box, and obtain the vector ;

[0094] Step 505: Implement bin mapping: divide the interval range [0,1] into d bins, and randomly generate elements in each bin according to the vector BN to obtain the same elements as the vector Vectors with the same number of elements , and the vector Map the elements in the vector BV one by one in order, restore the order of the elements in the vector BV according to the original index, and get the vector ;

[0095] Step 506: Upsampling: vector Upsample to generate an attention map consistent with the corresponding global view size hw×hw .

[0096] It should be noted that for step 5 to generate the attention map, the Kullback–Leibler divergence loss function is used to calculate the attention alignment loss The detailed calculation formula can be found in step 7.

[0097] Step 6: Mark local key areas and obtain local features:

[0098] First, according to the global information G generated in step 5 e , sort the attention weights of all image blocks in each global view in reverse order; through the position mask sampling method, selectively sample image blocks with high attention weights, that is, local key areas, according to the mask ratio sr, and record the positions of local key areas.

[0099] Then, different mask values ​​are used to mark the sampled image blocks and the unsampled image blocks, namely the local key areas and the local non-key areas, respectively, to generate a mask matrix with the same dimension as the global feature; at the feature level, the mask matrix is ​​multiplied element by element with the global feature to obtain a local feature that only retains the information of the local key areas. ; Among them, the global feature The generated local features are denoted as , by the global feature The generated local features are denoted as , by the global feature The generated local features are denoted as , by the global feature The generated local features are denoted as .

[0100] The details are as follows:

[0101] Step 601, sorting in reverse order: The global information consists of the attention weight value of each image block in the global view. These weight values ​​are arranged according to the position information of the corresponding image blocks to form a sequence, and are sorted in reverse order according to the size of the attention weight value.

[0102] Step 602, sampling: Calculate the sampling number according to the mask ratio, sample the attention weight values ​​in sequence until the sampling is completed, record the last sampled weight value and set it as the threshold.

[0103] Step 603, marking: in the global view, according to the position order of the image blocks, compare the threshold and the attention weight of each image block in turn. If the threshold is greater than the attention weight of a certain image block, the image block at this position is recorded as a local key area, otherwise, it is recorded as a local non-key area; according to the position order of the image blocks, mark each image block in turn. For the image block recorded as a local key area, the mask value of the position is marked as 1, otherwise, its mask value is marked as 0, and a mask matrix with the same dimension as the global feature is generated.

[0104] Step 604: Generate local features: multiply the mask matrix and the global features element by element to obtain local features that only retain information of the local key area. .

[0105] Sort in descending order according to the size of the attention weight; set the threshold according to the mask ratio sr, the number of samples is S=n×sr, and the threshold T is the weight value of the last sample in the list.

[0106] The threshold T is associated with the attention weight corresponding to each image block. For comparison, if the threshold T is greater than the attention weight of a certain image block , then the mask value of the image block Marked as 0; if the threshold T is less than the attention weight of a certain image block , then the mask value of the image block Marked as 1, get the label sequence, reconstruct the label sequence, and generate a mask matrix with the same dimension as the global feature ; Multiply the mask matrix and the global feature element by element to get the local feature. The generated local features are denoted as , by the global feature The generated local features are denoted as , by the global feature The generated local features are denoted as , by the global feature The generated local features are denoted as .

[0107] It should be noted that the local features obtained in step 6 are used to calculate the local discriminant loss The detailed calculation formula can be found in step 7.

[0108] The present invention also includes step 7, a loss calculation step. The total loss of the present invention includes global contrast loss, attention distillation loss and local discrimination loss.

[0109] (1) Calculate global contrast loss: Use the contrast loss function to calculate the global contrast loss for the global features extracted in step 3. . Global contrast loss The calculation formula is as follows:

[0110] ;

[0111] ;

[0112] ;

[0113] in, is a hyperparameter that controls the feature distribution, express middle The positive sample of express middle Negative samples of express middle The positive sample of express middle Negative samples.

[0114] (2) Calculate the attention distillation loss: Use the Kullback–Leibler divergence loss function to calculate the attention distillation loss on the attention map generated in step 5 Attention Distillation Loss The calculation formula is as follows:

[0115] ;

[0116] ;

[0117] .

[0118] (3) Calculate the local discriminant loss; use the cross entropy loss function to calculate the local discriminant loss for the local features obtained in step 6 . Calculate the local discriminant loss The calculation formula is as follows:

[0119] ;

[0120] ;

[0121] .

[0122] Total loss for learning global and local information It includes global contrast loss, attention distillation loss and local discrimination loss. The formula is:

[0123] ;

[0124] in, , , is the loss weight; during the optimization process, the AdamW optimizer is used to dynamically adjust the parameters of the model, but only the encoder Participate in the back propagation process of the network, and the momentum encoder Does not participate in the gradient back propagation process.

[0125] As another embodiment of the present invention, a self-supervised fine-grained feature learning system based on global and local information is also provided, which is used to implement the self-supervised fine-grained feature learning method based on global and local information as described above, and the system includes a global attention distillation module and a local key area learning module.

[0126] The global attention distillation module divides each view into multiple equal-sized but non-overlapping image blocks. Taking the image blocks as units, different methods are used to integrate global information for different network branches during global contrast learning. The global information obtained by different methods is mapped to the same contrast space and represented in the form of an attention map. The attention distillation loss is calculated. Under the constraint of the attention distillation loss, the learning of important features in the global information is emphasized.

[0127] The local key area learning module: evaluates the importance of each image block in the global view according to the global information, and selectively retains relatively important image blocks, i.e., local key areas, at the feature level according to the mask ratio through the position mask sampling method; strengthens the information of the local key areas under the constraint of the local discriminant loss, and learns fine-grained features from the local key areas.

[0128] The functional implementation and data processing process of each module can be found in the detailed steps of the self-supervised fine-grained feature learning method based on global and local information as recorded above, which will not be repeated here.

[0129] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Any changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A self-supervised fine-grained feature learning method based on global and local information, characterized in that: The following steps are involved: Step 1: Generate a global view: Input a batch of images, for each original image I in the same batch e , two different but equal-sized global views x are generated through two random enhancement methods e 1 and x e 2 , each global view is denoted as x e ; Step 2: Generate global view sample pairs: Two different global views generated from the same image but with different enhancement methods are considered as positive sample pairs, and two global views generated from different images and with different enhancement methods are considered as negative sample pairs; Step 3: Segment the global view and extract global features: Each global view x obtained in step 1 e Divide into n equal-sized but non-overlapping image blocks V e , all image blocks are flattened into sequences and linearly mapped into a D-dimensional space, using the Transformer-based encoder f q and by the encoder f q The momentum encoder f is generated by the momentum update of the parameters k For the global view x e 1 and x e 2 Encode and obtain global features and Step 4: Collect global information: Different methods are used to collect the global view x e 1 and x e 2 The global information G e 1 and G e 2 , each global view x e The global information is represented as G e ; In step 4, for the global view x e 1 , records the encoder f during the network back propagation process q The output feature map and output result of the last Transformer layer of the , by calculating the gradient to reflect the attention weight of each image block in the global view to the output result, the global information G of the entire view is obtained e 1 ; For the global view x e 2 , momentum encoder f k Each Transformer layer in the model relies on a multi-head self-attention mechanism to calculate the attention weights that reflect the relative relationship between each image block and other image blocks, collect and calculate the attention weights of all layers of the entire view, and obtain the global information G of the entire view. e 2 ; In step 4, for the global view x e 1 , records the encoder f during the network back propagation process q The output feature map and output result of the last Transformer layer of the , by calculating the gradient to reflect the attention weight of each image block in the global view to the output result, the global information G of the entire view is obtained e 1 ; For the global view x e 2 , momentum encoder f k Each Transformer layer in the model relies on a multi-head self-attention mechanism to calculate the attention weights that reflect the relative relationship between each image block and other image blocks, collect and calculate the attention weights of all layers of the entire view, and obtain the global information G of the entire view. e 2 ; Step 4: Collect global information as follows: For the view generated by enhancement method ①, the encoder f in the back propagation process of the global contrast network q The feature map and decision results of the last Transformer layer are used to generate the class activation map, which reflects the important information in the view. Specifically, for the encoder f q The output result y is back-propagated to calculate the gradient relative to the output feature map Y of the last Transformer layer Perform global average pooling on the gradient to obtain the weight vector W of each feature channel in the view. Multiply the weight vector W by the output feature map Y to obtain the attention weight G for the view. 1 ; For the view generated by enhancement method ②, the momentum encoder f k Each Transformer layer in the above example relies on the multi-head self-attention mechanism to focus on important information. Specifically, each Transformer layer calculates the attention weights between the vector Q of the query image block and the vectors K of other image blocks, and obtains the attention weight list A containing all layers, where each layer of the attention weight list contains the attention weights a corresponding to n image blocks. i ; Collect and calculate the attention weights of all layers of this view as G 2 ; Step 5: Generate attention map using global information: For global information G e 1 and global information C e 2 Different methods are used to generate the attention map H e 1 and H e 2 , each global view x e The attention map is represented as H e ; Step 5: The specific steps of generating the attention map using global information are as follows: For the view generated by enhancement method ①, the global information G 1 After normalization, we get H′ 1 , for H′ 1 Upsample to generate an attention map H that is consistent with the size of the corresponding global view 1 ; For the view generated by enhancement method ②, the global information G 2 Perform smoothing, binning, and upsampling to generate an attention map H that is consistent with the size of the corresponding global view 2 ; Step 6: Mark local key areas and obtain local features: According to the global information G e , sort the attention weights of all image blocks in each global view in reverse order; sample image blocks with high attention weights, i.e., local key areas, through the position mask sampling method, and record the positions of local key areas; generate a mask matrix, multiply the mask matrix by the global feature element by element, and obtain the local feature Z that only retains the information of the local key area e m ; Step 7, loss calculation: the total loss includes global contrast loss, attention distillation loss and local discrimination loss; Among them, global contrast loss: use the contrast loss function to calculate the global contrast loss of the global features extracted in step 3 Attention distillation loss: The attention distillation loss is calculated using the Kullback-Leibler divergence loss function on the attention map generated in step 5. Local discriminant loss: Use the cross entropy loss function to calculate the local discriminant loss for the local features obtained in step 6 Total loss The calculation formula is: Among them, α, β, and γ are loss weights; during the optimization process, the Adamw optimizer is used to dynamically adjust the parameters of the model, but only the encoder f q Participate in the back propagation process of the network, and the momentum encoder f k Does not participate in the gradient back propagation process.

2. The self-supervised fine-grained feature learning method based on global and local information according to claim 1, characterized in that: The specific steps of step 1 are as follows: During the training process, the batch size selected each time is E, and each original image I in the same batch e After random enhancement method ① and random enhancement method ②, two different but equal-sized global views x are generated. e 1 and the global view x e 2 ,The random enhancement method includes a combination of random cropping commanded by random parameters, Gaussian blur and horizontal folding.

3. The self-supervised fine-grained feature learning method based on global and local information according to claim 1, characterized in that: In step 2, set x1 1 is the global view generated by the original image I1 through enhancement method ①, x1 2 is the global view generated by the original image I1 through enhancement method ②, x1 1 and x1 2 are mutually positive sample pairs; set x2 2 is the global view generated by the original image I2 through enhancement method ②, x1 1 and x2 2 are mutually negative sample pairs; in the same batch, the view set U generated by enhancement method ① 1 Contains x e 2 The positive sample x e 1 And the negative sample set U 1 -{x e 1 }; Similarly, the view set U generated by the enhanced method ② 2 Contains x e 1 The positive sample x e 2 And the negative sample set U 2 -{x e 2 }.

4. The self-supervised fine-grained feature learning method based on global and local information according to claim 1, characterized in that: For step 6, the details are as follows: Step 601, sorting in reverse order: the global information is composed of the attention weight value of each image block in the global view. These weight values ​​are arranged according to the position information of the corresponding image blocks to form a sequence, and are sorted in reverse order according to the size of the attention weight value; Step 602, sampling: Calculate the sampling number according to the mask ratio, sample the attention weight values ​​in sequence until the sampling is completed, record the last sampled weight value and set it as the threshold; Step 603, marking: in the global view, according to the position sequence of the image blocks, the threshold and the attention weight of each image block are compared in turn. If the threshold is greater than the attention weight of a certain image block, the image block at this position is recorded as a local key area, otherwise, it is recorded as a local non-key area; according to the position sequence of the image blocks, each image block is marked in turn. For the image block recorded as a local key area, the mask value of the position is marked as 1, otherwise, its mask value is marked as 0, and a mask matrix with the same dimension as the global feature is generated; Step 604: Generate local features: multiply the mask matrix and the global features element by element to obtain local features Z that only retain information of the local key area. e m .

5. A self-supervised fine-grained feature learning system based on global and local information, characterized by: For implementing the self-supervised fine-grained feature learning method based on global and local information as described in any one of claims 1 to 4, the system comprises a global attention distillation module and a local key area learning module, The global attention distillation module divides each view into multiple equal-sized but non-overlapping image blocks. In the process of global contrastive learning, different methods are used for different network branches to integrate global information, and the global information obtained by different methods is mapped to the same contrast space, represented in the form of an attention map, and the attention distillation loss is calculated. Under the constraint of the attention distillation loss, the learning of important features in the global information is emphasized. The local key area learning module: evaluates the importance of each image block in the global view according to the global information, and selectively samples the image blocks with high attention weights, i.e., the local key areas, at the feature level according to the mask ratio through the position mask sampling method; strengthens the information of the local key areas under the constraint of the local discriminant loss, and learns fine-grained features from the local key areas.