A multi-network contrastive learning method based on global and local features

Abstract

The application belongs to the field of self-supervised learning visual representation, and discloses a multi-network contrast learning method based on global and local features, image data is collected, three perturbation views are obtained by the same series of data enhancement for each image in the training set; the images are respectively input into three different networks to extract feature representation; the output of the online network and the output of the other two networks are respectively calculated contrast loss and combined into total loss; the online network is updated by gradient through minimizing the total loss; the trained network is used for image classification of different data sets, and the classification accuracy is used to evaluate the performance of the proposed method. The application introduces global and local feature information in self-supervised contrast learning, improves the accuracy of image classification, enhances the generalization performance of the deep neural network, at the same time, the number of samples used for contrast is expanded, and the training efficiency of the proposed model is improved.

Description

Technical Field

[0001] This invention belongs to the field of self-supervised learning visual representation, specifically involving a multi-network contrastive learning method based on global and local features. Background Technology

[0002] Deep learning is a subfield of machine learning that processes and analyzes data by mimicking the neural network structure of the human brain. In recent years, with the improvement of computing power and the accumulation of massive amounts of data, deep learning has made significant progress in areas such as image recognition, speech recognition, and natural language processing, becoming a hot topic and frontier in artificial intelligence research. Since the beginning of the 21st century, the rapid improvement of computing power and the widespread adoption of the internet have enabled deep learning to flourish.

[0003] However, training deep neural network models typically requires large amounts of labeled data, which is costly to acquire. Self-supervised learning, a research focus in deep learning, has gained popularity because it uses large amounts of unlabeled data to train models. Its advantage lies in learning useful representations without manual annotation. Self-supervised learning generates pseudo-labels based on the internal structure of the data to guide model learning, fully leveraging the information contained within large datasets. These self-supervised learning methods often involve defining a pre-defined task that can be solved using only the data itself, training the model to solve this task, and learning features helpful for downstream tasks.

[0004] As a common self-supervised learning method, contrastive learning has shown encouraging results in visual tasks. It uses instance discrimination as a pre-defined task for training the model. Typically, two augmented views of images obtained from two data augmentations are used as the positive pair, while the negative sample is a view of another image obtained through data augmentation. The pre-defined task is accomplished by bringing the positive pair closer together and pushing the negative sample further apart in the feature space. During the training process of the model to complete the pre-defined task, the model can learn features that are truly useful for downstream tasks.

[0005] However, most existing contrastive learning methods only use image-level feature information, ignoring the correlation between global and local features. Local feature information is often crucial in dense prediction tasks because these tasks rely on models operating at the pixel or image patch level. If only global feature information is used, the model struggles to learn the features necessary to perform these tasks. For example, semantic segmentation aims to associate each pixel in an image with a specific semantic label, while object detection aims to determine the location and category of different objects in an image. Therefore, incorporating local feature information is beneficial for models to perform these pixel-level or image patch-level prediction tasks. However, this does not mean that local feature information is no longer important in downstream tasks such as image-level prediction. Since local feature information can enhance model performance in dense prediction tasks, it can help the model learn information that cannot be noticed when using only global feature information. Therefore, better methods are needed to comprehensively utilize both global and local feature information.

[0006] To address the aforementioned issues, numerous studies have explored the use of image patches in self-supervised learning, such as using a jigsaw puzzle as a pre-defined task to divide a portion of an image into small patches. After these patches are shuffled, the model predicts their original arrangement. In this process, the network can learn image structure or texture features. Jigsaw clustering randomly combines image patches from multiple images into a new image. The encoder is then trained by clustering given image patches and reconstructing the original image. On the other hand, many self-supervised learning works using transformers as encoders also leverage image patches for model training. Drawing on experience in natural language processing, ViT divides images into patches to match the input requirements of transformers. In MAE, certain regions in an image are masked, and the model is trained to reconstruct pixel information within these masked patches. Similarly, BEIT randomly masks some image patches and predicts their corresponding visual labels. By completing this Masked Image Modeling (MIM) task, the model can learn features in the image. These studies have also recognized the crucial role of local features in model training. However, most of these methods use image patches to design pre-defined tasks rather than to introduce local feature information. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides a multi-network contrastive learning method based on global and local features, which is applied to the field of image classification. It uses multiple neural networks with different structures to extract global and local features of a given image, and associates the global and local feature information of the image by minimizing the total loss. This allows the proposed model to comprehensively learn feature information at different scales of the image, thereby significantly improving the image classification accuracy and training efficiency of the proposed model.

[0008] The present invention discloses a multi-network contrastive learning method based on global and local features, comprising the following steps:

[0009] Step 1: Collect image data, divide it into training set and test set, and obtain three perturbation views for each image in the training set through the same series of data augmentation.

[0010] Step 2: Input any perturbation view into the online network to extract the anchor point feature representation of the image;

[0011] Step 3: Input the second perturbation view into the neighbor sample network to extract the global feature representation of the image. At the same time, the neighbor sample network constructs a support set to replace the feature representation output by the network with a neighbor feature representation.

[0012] Step 4: After segmenting the last perturbation view according to the specified rules, it is then input into the segmentation sample network to extract local feature representations of the image. At the same time, the segmentation sample network will combine several feature representations extracted by the encoder according to the specified rules.

[0013] Step 5: Calculate and compare the output of the online network with the outputs of the other two networks, and then combine them into a total loss;

[0014] Step 6: Update the parameters of the online network by minimizing the total loss value, while not updating the parameters of the other two networks.

[0015] Step 7: Use the trained network to classify images on different datasets, and calculate the top 1 and top 5 classification accuracies to evaluate the performance of the proposed method.

[0016] Furthermore, in step 2, the anchor point feature representation is extracted, specifically as follows:

[0017] The online network consists of a backbone encoder f o Projector g o It consists of a predictor q;

[0018] In an online network, perform random data augmentation on the original image x input to the network to obtain an augmented view x. o Based on view x o Main encoder f o and projector g o Output z o =g o (f o (x o Then z o The output p = q(z) is passed to the predictor q to generate the online network. o ).

[0019] Furthermore, in step 3, the global feature representation is extracted, specifically as follows:

[0020] The neighboring sample network contains only the backbone encoder f m and projector g m ;

[0021] In a neighbor-samples network, the same series of random data augmentations are performed on the original image x input to the network to obtain an augmented view x. m Based on view x m Main encoder f m and projector g m Output a representation of z′ m =g m (f m (x m ));

[0022] A support set, implemented using a queue, is constructed to store the latest feature representations. To better extract rich global feature representations, feature representation z′ is selected from the support set. m nearest neighbor z m As the output of this network.

[0023] Furthermore, in step 4, the local feature representation is extracted, specifically as follows:

[0024] The segmentation sample network contains only one backbone encoder f s and a projector g s ;

[0025] In the segmentation sample network, the same series of random data augmentations are performed on the original image x input to the network to obtain the augmented view x. s ;

[0026] To incorporate local feature information for self-supervised contrastive learning, the enhanced view x will be... s The image is divided into 2×2 blocks; then the image blocks are fed into the backbone encoder f. s At the same time, the main encoder f s The outputs are combined with each other to obtain S combined outputs, where S is calculated according to the following formula:

[0027]

[0028] Where k∈{1,2,3,4} is the number of image patches involved in the combination, and C represents the combination that satisfies the condition and... Each combination outputs v j It is achieved by calculating the selected k backbone encoders f sThe average value of the output is obtained; j is the index of the combined output, with a value range of [1, S]; therefore, the combined output set v = {v} is obtained. j |j∈{1,...,S}}, and feed it into the projector g. s In the middle, each combination outputs v j The corresponding output is a feature representation c. j Ultimately, the output of this network is a set of feature representations c = {c j |j∈{1,...,S}}.

[0029] Furthermore, step 5 specifically involves:

[0030] Contrast loss function between online network and neighboring sample network The definition is as follows:

[0031]

[0032] Where p and z m These are the outputs of the online network and the neighbor sample network, respectively. τ is the uniform temperature hyperparameter, and N... m It is z m The corresponding set of negative samples, It is N m One negative sample;

[0033] Comparative loss functions between online networks and segmentation sample networks The definition is as follows:

[0034]

[0035] Where S is the number of combined outputs in the segmentation sample network, and p and c j These are the outputs of the online network and the segmented sample network, respectively. τ is the uniform temperature hyperparameter, and N... s It is C j The corresponding set of negative samples, It is N s One negative sample; finally, the overall contrastive loss function is defined as follows:

[0036]

[0037] Where λ>0 is the equilibrium hyperparameter.

[0038] Furthermore, in step 6, the parameter update methods differ for different networks;

[0039] The parameters of the online network are updated through backpropagation;

[0040] The backbone encoder and projector parameters of the neighboring sample network are updated using momentum:

[0041]

[0042]

[0043] in and These are the parameters of the backbone encoder and projector in the neighboring sample network. and These are the parameters of the backbone encoder and projector in the online network, where m is the momentum coefficient.

[0044] The parameters of the backbone encoder and projector in the segmentation sample network are the same as those in the online network:

[0045]

[0046]

[0047] in and These are the parameters of the backbone encoder and projector in the segmentation sample network. and These are the parameters corresponding to those in an online network.

[0048] The beneficial effects of this invention are as follows: Considering that self-supervised contrastive learning in computer vision generally neglects the correlation between global and local image features and lacks a combination of global and local information, this invention proposes a multi-network contrastive learning method based on global and local features. This method uses multiple neural networks with different structures to extract global and local features of a given image, calculates the contrastive loss between these features, and by minimizing the total loss, correlates the global and local feature information of the image, enabling the proposed model to comprehensively learn feature information at different scales of the image. In this way, this invention not only enhances the generalization performance of deep neural networks in image classification but also provides more training samples during the contrastive learning process, thereby significantly improving the image classification accuracy and training efficiency of the proposed model. Attached Figure Description

[0049] Figure 1 This is a flowchart of the method described in this invention;

[0050] Figure 2 This is a schematic diagram of the algorithm framework of the present invention;

[0051] Figure 3 This is a schematic diagram illustrating the method described in this invention and the t-SNE visualization performed on the CIFAR-10 dataset using MoCo v2. Detailed Implementation

[0052] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0053] like Figure 1 As shown, this invention describes a multi-network contrastive learning method based on global and local features, denoted as MNCLGLF. The training sets used are CIFAR10, CIFAR100, and Tiny ImageNet, and the convolutional neural network used is ResNet-18. The method includes the following steps:

[0054] Step 1: Collect image data, divide it into training and test sets, and obtain three perturbation views by performing data augmentation on the image x in the training set using the same series of data;

[0055] Step 2: Select one of the perturbation views x o The input is fed into an online network to extract the anchor point feature representation p of the image;

[0056] Step 3: Move the second perturbation view x m The input is fed into the neighbor sample network to extract global feature representations of the image. Simultaneously, the neighbor sample network constructs a support set to process the feature representation z′ output by the network. m Replace with a neighboring feature representation z m ;

[0057] Step 4: Move the last perturbation view x s After being segmented according to the specified rules, the data is then input into the segmentation sample network to extract local feature representations of the image. At the same time, the segmentation sample network will combine several feature representations extracted by the backbone encoder according to the specified rules.

[0058] Step 5: Combine the output p of the online network with the outputs z of the other two networks. m and c = {c j Calculate the contrastive loss for each |j∈{1,...,S}} and combine them into the total loss;

[0059] Step 6: Update the parameters of the online network by minimizing the total loss value, while not updating the parameters of the other two networks.

[0060] Step 7: Use the trained network to classify images on different datasets, and calculate the top 1 and top 5 classification accuracies to evaluate the performance of the method.

[0061] like Figure 2 As shown in the diagram, the algorithm framework of a multi-network contrastive learning method based on global and local features according to the present invention includes the following parts:

[0062] Data processing section: Image data is acquired, divided into training and test sets, and the image x in the training set is augmented using the same series of data to obtain three perturbation views x. o x m and x s ;

[0063] The convolutional neural network update section includes three types of networks: online networks, nearest neighbor networks, and segmentation networks. The online network consists of a backbone encoder f. o Projector g o It consists of a predictor q. The parameters of this network are updated via backpropagation. The neighbor sample network contains only the backbone encoder f. m and projector g m The backbone encoder and projector parameters of this network are updated using momentum. The segmentation sample network also contains only one backbone encoder f. s and a projector g s The parameters of the backbone encoder and projector in this network are the same as those in the online network.

[0064] Convolutional Neural Network Structure: In online networks, based on augmented view x o The main encoder and projector output z o =g o (f o (x o The predictor q generates the network's output p = q(z). o In the neighbor sample network, based on the enhanced view x m The main encoder and projector output z′ m =g m (f m (x m )), and use the representation z′ m nearest neighbor z m As the output of this network, the segmented sample network will enhance the view x. s The image is divided into 2×2 blocks and then fed into the encoder. Simultaneously, the backbone encoder f... s The outputs are combined with each other to obtain a combined output set v = {v j |j∈{1,...,S}}, where each combination outputs v j This is obtained by calculating the average value of the selected encoder output, where j is the combined output number, ranging from [1, S]. The combined output set v is then fed into the projector g. s In the middle, each combination outputs v j The corresponding output is a feature representation c. j Ultimately, the network's final output is a set of feature representations c = {c j|j∈{1,...,S}}.

[0065] The method described in this invention comprises the following parts:

[0066] (1) Image feature extraction

[0067] 1. Online Network: The online network consists of the backbone encoder f o Projector g o It consists of a predictor q and a first enhanced view x. o It is obtained by performing a single random data augmentation on a given image x, i.e., x o =t(x). Based on this view, the trunk encoder and projector output z o =g o (f o (x o Then we will z o The features are passed to the predictor q to produce the feature representation p = q(z) o The parameters of this network branch are updated via backpropagation.

[0068] 2. Neighborhood Sample Network: The neighborhood sample network only contains the backbone encoder f m and projector g m Enhanced view x m It is obtained by performing the same series of random data augmentations on a given image x, i.e., x m = t′(x). Based on the enhanced view x m The main encoder and projector output z′ m =g m (f m (x m )), and use the representation z′ m nearest neighbor z m As the output of this network, the backbone encoder and projector parameters are updated using momentum:

[0069]

[0070]

[0071] in and These are the parameters of the backbone encoder and projector in the neighboring sample network. and These are the parameters of the backbone encoder and projector in the online network, where m is the momentum coefficient. To better extract rich global feature representations, this invention uses the feature representation z′. m nearest neighbor z m As the output of this network, this invention constructs an additional memory structure to find the nearest neighbor z.m Specifically, it is a queue used to maintain a set of feature representations that approximate the distribution of the dataset. This queue is randomly initialized and updated during the training phase. At the end of each training step, the oldest batch of feature representations is removed from the queue, and the current batch is enqueued. The size of the queue needs to be set to a sufficiently large value to allow the feature representations in the queue to approximate the true data distribution.

[0072] 3. Segmentation Sample Network: The segmentation sample network also contains only one backbone encoder f. s and a projector g s Enhanced view x s It is obtained by performing the same series of random data augmentations on a given image x, i.e., x s =t″(x). The parameters of the backbone encoder and projector of this network are the same as those in the online network:

[0073]

[0074]

[0075] in and These are the parameters of the backbone encoder and projector in the segmentation sample network. and These are the corresponding parameters in the online network. To incorporate local feature information for self-supervised contrastive learning, the enhanced view x will be... s The image is divided into 2×2 blocks. These image blocks are then fed into the backbone encoder f. s At the same time, the main encoder f s The outputs are combined with each other to obtain S combined outputs, where S is calculated according to the following formula:

[0076]

[0077] Where k∈{1,2,3,4} is the number of image patches involved in the combination, and C represents the combination that satisfies the condition and... Each combination outputs v j It is achieved by calculating the selected k backbone encoders f s The average value of the output is obtained, where j is the index of the combined output, ranging from [1, S]. Therefore, we obtain the combined output set v = {v...} j |j∈{1,...,S}}, and feed it into the projector g. s In the middle, each combination outputs v j The corresponding output is a feature representation c. j Ultimately, the output of this network is a set of feature representations c = {c j |j∈{1,...,S}}.

[0078] (2) Loss Function

[0079] 1. Loss Calculation: Comparative Loss Function between Online Network and Neighboring Sample Networks The definition is as follows:

[0080]

[0081] Where p and z m These are the outputs of the online network and the neighbor sample network, respectively. τ is the uniform temperature hyperparameter, and N... m It is z m The corresponding set of negative samples, It is N m One of the negative samples.

[0082] Comparative loss functions between online networks and segmentation sample networks The definition is as follows:

[0083]

[0084] Where S is the number of combined outputs in the segmentation sample network, and p and c j These are the outputs of the online network and the segmented sample network, respectively. τ is the uniform temperature hyperparameter, and N... s It is C j The corresponding set of negative samples, It is N s One of the negative samples.

[0085] 2. Combination of Loss Functions: Finally, the overall comparative loss function is defined as follows:

[0086]

[0087] Where λ>0 is the equilibrium hyperparameter.

[0088] The datasets involved in this invention are described in Table 1, including dataset name, number of categories, image size, and the number of samples in the training and test sets. The top-1 and top-5 accuracies of this invention compared to other classic self-supervised methods on three benchmark image datasets are shown in Table 2. Top-1 and top are commonly used evaluation metrics for image classification. Top-1 means that the correct category is predicted by the network model, and top-5 means that the correct category is included among the top five predicted categories.

[0089] Table 1. Description of the dataset involved in this invention.

[0090] CIFAR-10 10 32x32 50000 10000 CIFAR-100 100 32x32 50000 10000 Tiny ImageNet 200 64x64 100000 10000

[0091] Table 2. Top-1 and top-5 accuracy of different methods on three benchmark image datasets.

[0092]

[0093] The best results are shown in bold. Due to experimental limitations, the batch size was set to 256, the extra queue size for all methods was 16384, and the temperature hyperparameter was set to 1. This invention uses top-1 accuracy and top-5 accuracy to evaluate the performance of all methods on these three benchmark datasets. The method MNCLGLF described in this invention outperforms other classic self-supervised contrastive learning methods on the three benchmark datasets. The combination of global and local feature information can help the network model obtain more salient representations and enhance the generalization ability of the network model. However, compared with supervised methods, the method MNCLGLF still lags behind in accuracy.

[0094] Figure 3 This diagram illustrates the t-SNE visualization of the MNCLGLF method and MoCo v2 on the CIFAR-10 dataset, showcasing the semantic features obtained by these two methods. MoCo v2 fails to separate categories 2 through 6 in the latent feature space, and the boundaries between other categories are also relatively blurry. The MNCLGLF method exhibits only relatively blurry distinctions between two categories, but compared to MoCo v2, it shows clear boundaries between other categories and maintains compact embeddings within each category. Therefore, the MNCLGLF method of this invention has better generalization performance and can obtain semantic features that are truly useful for downstream tasks.

[0095] The above description is merely a preferred embodiment of the present invention and is not intended to further limit the present invention. All equivalent changes made based on the description and drawings of the present invention are within the protection scope of the present invention.

Claims

1. A multi-network contrastive learning method based on global and local features, characterized in that, Includes the following steps: Step 1: Collect image data, divide it into training set and test set, and obtain three perturbation views by augmenting each image in the training set with the same series of data. Step 2: Input any perturbation view into the online network to extract the anchor point feature representation of the image; Step 3: Input the second perturbation view into the neighbor sample network to extract the global feature representation of the image. At the same time, the neighbor sample network constructs a support set to replace the feature representation output by the network with a neighbor feature representation. Step 4: After segmenting the last perturbation view according to the specified rules, it is then input into the segmentation sample network to extract local feature representations of the image. At the same time, the segmentation sample network will combine several feature representations extracted by the encoder according to the specified rules. Step 5: Calculate and compare the output of the online network with the outputs of the other two networks, and then combine them into a total loss; Step 6: Update the parameters of the online network by minimizing the total loss value, while not updating the parameters of the other two networks. Step 7: Use the trained network to classify images on different datasets, and calculate the top 1 and top 5 classification accuracies to evaluate the performance of the proposed method. In step 3, the global feature representation is extracted, specifically as follows: The neighboring sample network contains only the backbone encoder. and projector ; In a neighbor-sample network, the original image is the input to the network. Perform random data augmentation on the same series to obtain an augmented view. Based on view Main encoder and projector Output a representation ; Construct a support set implemented by a queue, which stores several of the latest feature representations; select feature representations from the support set. nearest neighbor As the output of this network; In step 4, the local feature representation is extracted, specifically as follows: The segmentation sample network contains only one backbone encoder. and a projector ; In a segmentation sample network, the original image is input to the network. Perform random data augmentation on the same series to obtain an augmented view. ; To incorporate local feature information for self-supervised contrastive learning, the enhanced view will be... The image is divided into 2×2 blocks; then the image blocks are fed into the backbone encoder. At the same time, the main encoder The outputs are combined with each other to obtain A combination output, in which Calculate using the following formula: ， in It is the number of image blocks involved in the combination. Indicates that the combination satisfies and Each combination output It is through calculation of the selected One main encoder Obtained by averaging the output; It is the combined output number, and its value range is... Therefore, the combined output set was obtained. Send it to the projector In the middle, each combination output The corresponding output is a feature representation. Ultimately, the output of this network is a set of feature representations. ; Step 5 specifically involves: Contrast loss function between online network and neighboring sample network The definition is as follows: ， in and These are the outputs of the online network and the neighboring sample network, respectively. It is a unified temperature hyperparameter. yes The corresponding set of negative samples, yes One negative sample; Comparative loss functions between online networks and segmentation sample networks The definition is as follows: ， in, This represents the number of combined outputs in the segmentation sample network. and These are the outputs of the online network and the segmentation sample network, respectively. It is a unified temperature hyperparameter. yes The corresponding set of negative samples, yes One negative sample; finally, the overall contrastive loss function is defined as follows: ， in It is a balancing hyperparameter.

2. The multi-network contrastive learning method based on global and local features according to claim 1, characterized in that, In step 2, the anchor point feature representation is extracted, specifically as follows: The online network consists of a backbone encoder. Projector and predictor composition; In an online network, the raw image input to the network. Perform a random data augmentation to obtain an augmented view Based on view Main encoder and projector Output Then Passed to the predictor Generate output of online network .

3. The multi-network contrastive learning method based on global and local features according to claim 1, characterized in that, In step 6, the parameter update methods differ for different networks; The parameters of the online network are updated through backpropagation; The backbone encoder and projector parameters of the neighboring sample network are updated using momentum: ， ， in and These are the parameters of the backbone encoder and projector in the neighboring sample network. and These are the parameters of the backbone encoder and projector in an online network. It is the momentum coefficient; The parameters of the backbone encoder and projector in the segmentation sample network are the same as those in the online network: ， ， in and These are the parameters of the backbone encoder and projector in the segmentation sample network. and These are the parameters corresponding to those in an online network.

Images