Method and device for improving contrastive learning by small batch sampling

Abstract

The application discloses a method and device for improving contrast learning through small batch sampling, wherein the method comprises the following steps: obtaining an instance dataset, and constructing a proximity graph by using the instance dataset; performing preset number sampling on instances in the instance dataset on the proximity graph to obtain a preset number of training data; wherein the training data are used for learning embedding representation of the instances; inputting the training data into multiple network models to perform feature extraction, and obtaining training instance features; training the multiple network models by using the training instance features and a loss function, and updating model parameters by using an optimizer to obtain trained multiple network models. Similar instances connected on the whole dataset can better explore global hard negative samples by sampling on the proximity graph. Significant performance improvement is presented on three modalities, which proves the effectiveness and universality of the sampling method, and the sampling method can also be combined with other models to improve model performance.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a method and apparatus for improving contrastive learning through mini-batch sampling. Background Technology

[0002] Batch contrastive learning has become a state-of-the-art self-supervised learning solution. Its core principle is to bring semantically similar instances closer together while separating different instances in mini-batches. However, the mini-batch negative sampling strategy is limited by the batch size and cannot prioritize information-rich negative samples (i.e., hard negative samples) globally.

[0003] Contrastive learning has long been a dominant approach in self-supervised representation learning, with applications in numerous fields, such as MoCo and SimCLR in computer vision, GCC and GraphCL in graph representation learning, and SimCSE in natural language processing. These important works in contrastive learning typically follow or slightly modify the framework of batch contrastive learning, as follows:

[0004]

[0005] Where {x1,...,xB} are typically mini-batch samples randomly loaded from dataset D. x generated through data augmentation i The encoder f(·) maps different data-enhanced versions of the same instance (positive sample pairs) to similar embedding representations and different instances in mini-batches (negative sample pairs) to different embedding representations. Batch contrastive learning is efficient, distinguishing all B(B-1) pairs of instances in a mini-batch and encoding only once for each instance. Its simplicity and efficiency have made it more popular than pairwise or triple-based methods, gradually becoming the dominant framework for contrastive learning. However, the performance of batch contrastive learning is closely related to batch size. Achieving a large (equivalent) batch size under limited computational and memory budgets is the goal of many important contrastive learning methods. For example, Memory Bank stores encoded embeddings from previous mini-batches as additional negative samples, while MoCo improves the consistency of stored negative samples through a momentum encoder. SimCLR shows that simply increasing the batch size of intra-batch contrastive learning to 8,192 is better than previously carefully designed methods. Despite these works highlighting the importance of batch size, another question arises—which instances in the mini-batch contribute the most?

[0006] Hard negative instances contribute the most—many works have provided strong support for a clear answer to this problem in very similar scenarios, especially in (graph-based) recommender systems. An intuitive explanation is that easily distinguishable negative sample pairs... The value becomes very small after early training, so hard negative samples contribute most of the loss and gradient. Hard negative sampling has achieved great success in many practical applications, such as improving Facebook search recall by 8%. The only significant difference between contrastive learning and recommendation systems is whether positive sample pairs are different augmented versions of the same instance or items purchased together.

[0007] In recent years, negative sampling techniques in contrastive learning have also made some progress. Previous research on negative sampling in contrastive learning has mainly focused on creating or enhancing negative sample pairs in a given mini-batch, and can be roughly divided into two categories: (1) reweighted contrastive loss, which modifies the weights of negative samples in the original InfoNCE loss. For example, DCL and HCL apply adjustable weight distributions to mitigate the false negative problem or amplify the impact of hard negative samples. (2) creating new negative samples, which details strategies for generating hard negative samples from candidates. For example, MoCHI suggests mixing candidate hard negative samples at the feature level to generate more challenging negative samples.

[0008] Existing methods directly use randomly sampled data of a batch size for training, namely RandSampler. Summary of the Invention

[0009] The present invention aims to at least partially solve one of the technical problems in the related art.

[0010] To this end, this invention proposes a novel mini-batch sampling ProSampler. Therefore, the objective of this invention is to implicitly add more hard negative samples in each mini-batch to significantly improve contrastive learning. To this end, this invention proposes a method for improving contrastive learning through mini-batch sampling.

[0011] To achieve the above objectives, this invention proposes a method for improving contrastive learning through mini-batch sampling, comprising:

[0012] Obtain an instance dataset and construct a proximity graph using the instance dataset;

[0013] A preset number of instances in the instance dataset are sampled on the neighborhood graph to obtain a preset number of training data; wherein, the training data is used to learn the embedding representation of the instances;

[0014] The training data is input into various network models for feature extraction to obtain training instance features;

[0015] The various network models are trained using the features of the training instances and the loss function, and the optimized model parameters are updated to obtain the trained network models.

[0016] In addition, the method for improving contrastive learning through mini-batch sampling according to the above embodiments of the present invention may also have the following additional technical features:

[0017] Furthermore, in one embodiment of the present invention, the step of constructing a proximity graph using the instance dataset includes: sampling multiple instances from the instance dataset using random sampling to obtain an instance candidate set; querying the instance candidate set for multiple nearest-neighbor instance samples of the instance in the embedding space; and constructing a proximity graph based on the multiple nearest-neighbor instance samples.

[0018] Furthermore, in one embodiment of the present invention, the step of sampling a preset number of instances in the instance dataset on the neighborhood graph to obtain a preset number of training data includes: starting from an instance node on the neighborhood graph, using a sampler to iteratively return to the starting point with a preset probability, or moving to a neighboring node at the current position with a probability proportional to the edge weight; until a batch of data satisfying the preset number of instance nodes is collected to obtain the preset number of training data.

[0019] Furthermore, in one embodiment of the present invention, the neighbor graph is represented as follows:

[0020]

[0021] Where V = {v1, ..., vn} represents instances in the dataset. These are node pairs generated through similarity metrics, i.e., edges in the neighborhood graph.

[0022] Furthermore, in one embodiment of the present invention, the expression for constructing a proximity graph based on the plurality of nearest neighbor instance samples is:

[0023]

[0024] Where K is the instance sample, e m Let N be the candidate set of instances. vi Indicates instance v in the neighborhood graph i The neighbor of , · is the inner product operation.

[0025] Furthermore, in one embodiment of the present invention, the expression for the loss function is:

[0026]

[0027] Among them, z iand z j It is a pair of positive samples, z k τ is the negative sample, B is the batch size, Sim(·) is the cosine similarity, and τ is the temperature coefficient.

[0028] To achieve the above objectives, another aspect of the present invention provides an apparatus for improving contrastive learning through mini-batch sampling, comprising:

[0029] The neighbor graph construction module is used to obtain an instance dataset and construct a neighbor graph using the instance dataset.

[0030] The training data acquisition module is used to sample a preset number of instances in the instance dataset on the neighborhood graph to obtain a preset number of training data; wherein, the training data is used to learn the embedding representation of the instances;

[0031] The feature extraction module is used to input the training data into various network models for feature extraction to obtain training instance features;

[0032] The module optimization module is used to train the various network models using the features of the training instances and the loss function, and to update the model parameters using an optimizer to obtain the trained various network models.

[0033] The apparatus for improving contrastive learning through mini-batch sampling in this invention utilizes a neighborhood graph to construct mini-batch data with hard samples, thereby improving the performance of contrastive learning.

[0034] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0035] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0036] Figure 1 This is a flowchart illustrating a method for improving contrastive learning through mini-batch sampling according to an embodiment of the present invention.

[0037] Figure 2 This is a flowchart of an algorithm for improving contrastive learning through mini-batch sampling according to an embodiment of the present invention.

[0038] Figure 3 This is a flowchart of a random walk algorithm with restart according to an embodiment of the present invention;

[0039] Figure 4 This is a diagram illustrating the overall framework of the ProSampler for small-batch sampling according to an embodiment of the present invention.

[0040] Figure 5 This is a schematic diagram of an apparatus for improving comparative learning through small-batch sampling according to an embodiment of the present invention. Detailed Implementation

[0041] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0042] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0043] The method and apparatus for improving contrastive learning through small-batch sampling according to embodiments of the present invention are described below with reference to the accompanying drawings.

[0044] The algorithm flow of this invention is as follows: Figure 2 As shown, ProSampler acts as a mini-batch sampler and can be naturally inserted into any batch contrastive learning method, such as SimCLR, SimCSE, and GraphCL. Specifically, during training, ProSampler first constructs a neighborhood graph using DkNN, updates the neighborhood graph after s training steps, and then randomly selects a starting node and samples a mini-batch on the neighborhood graph using RWR.

[0045] Figure 1 This is a flowchart of a method for improving comparative learning through small-batch sampling, according to an embodiment of the present invention.

[0046] like Figure 1 As shown, the method includes, but is not limited to, the following steps:

[0047] S1. Obtain the instance dataset and construct the neighborhood graph using the instance dataset.

[0048] Understandably, in order to construct a neighborhood graph utilizing hard negative samples, embodiments of the present invention connect nodes to other indistinguishable instances (i.e., hard ones) because they are close in the embedding space. To achieve the neighborhood graph, the k-nearest neighbor (kNN) method is used. kNN directly selects the k closest candidates in the embedding space for the query instance.

[0049] However, directly using kNN sampling to construct the neighborhood graph leads to two serious problems:

[0050] High time complexity: To retrieve the k closest instances, kNN will perform explicit comparisons between all instance pairs, resulting in a time complexity of O(N^2). 2 d), where N is the number of instances and d is the dimension of the embedding representation. For large-scale datasets, its computational complexity is prohibitive.

[0051] False Negative (FN) Problem: A false negative (FN) is defined as all negative instances in a batch that belong to the same class as the query instance. In self-supervised training, instances of the same class tend to cluster together in the embedding space, which increases the probability of selecting an FN.

[0052] To address these limitations, this invention proposes a debiased k-nearest neighbor (DkNN) approach to construct the neighborhood graph. In the training dataset, there are N instances {v} i |i=1,...,N} and the corresponding embedding representation {e} generated by the current encoder f i |i=1,...,N}. Unlike kNN, which directly selects the k nearest nodes, DkNN first randomly samples M (M<<N) neighbor candidates for each node, forming a candidate set E={e m Then, kNN is performed on the candidate set E to select neighbors. Randomly selected candidate sets help reduce computational complexity, specifically O(NMd). Furthermore, selecting neighbors of a finite size from the candidate set limits the number of false negatives.

[0053] As an example, such as Figure 4 As shown, to improve negative sampling techniques in contrastive learning by including hard samples in small batches of data, this invention first constructs a neighborhood graph. The nodes of the neighborhood graph are the instances themselves in the dataset, and the edges of the graph are constructed using the similarity learned between instances, which can be formalized as:

[0054]

[0055] Where V = {v1, ..., vn} represents instances in the dataset. These are node pairs generated through a similarity metric, i.e., edges in the graph. From the perspective of the neighborhood graph, the original mini-batch sampling can be considered as sampling a set of nodes on the neighborhood graph, which suggests that mini-batch data with hard negative samples should contain a large number of similar nodes that are close in the embedding space.

[0056] Next, the proposed k-nearest neighbor (kDNN) method for constructing a proximity graph will be described in detail. Specifically, firstly, M instances are randomly sampled from the dataset to form a candidate set E = {em}, where M is much smaller than the total number of instances N in the entire dataset. Then, kNN is performed on the candidate set E to select the k nearest neighbors to construct the proximity graph.

[0057]

[0058] Where, N vi Indicates instance v in the neighborhood graph i The neighbors are selected, and · is the inner product operation. Randomly selected candidate sets can help reduce computational complexity, with a time complexity of O(NMd). Furthermore, selecting neighbors of a finite size from the candidate set can limit the number of false negatives.

[0059] S2, sample a preset number of instances in the instance dataset on the neighborhood graph to obtain a preset number of training data; wherein, the training data is used to learn the embedding representation of the instances.

[0060] It is understandable that neighbor graph sampling involves performing mini-batch sampling on the neighbor graph G, obtaining a batch of similar nodes by sampling within local neighborhoods in the neighbor graph. This invention employs a theoretically guaranteed method: Random Walk with Restart (RWR). The algorithm is as follows... Figure 3 As shown, starting from a node, the sampler iteratively returns to the starting point with probability α, or moves to a neighbor of the current position with a probability proportional to the edge weight. This process continues until it has collected a fixed number of vertices. The key to using RWR is that it allows the sampler to flexibly balance the sampling probability within the restricted neighborhood by adjusting α, thus allowing the sampler to balance false negatives and hard negative samples.

[0061] As an example, after constructing the neighborhood graph, this invention utilizes a random walk with restart (RWR) to sample a specific instance on the neighborhood graph to obtain batch-sized training data. Specifically, starting from a node, the sampler iteratively returns to the starting point with probability α, or moves to a neighbor of the current position with a probability proportional to the edge weight. This process continues until it has collected a fixed number of vertices. The key to using RWR is that it allows the sampling probability within a restricted neighborhood to be tuned by adjusting α, thus allowing the sampler to flexibly balance the issues of false negatives and hard negative samples.

[0062] Understandably, the use of RWR to obtain a batch size of training data in this invention is supported by a theory. The theory proves that other nodes within a node's nearest neighbor region are highly similar.

[0063] Furthermore, mini-batch training data can be obtained to learn the embedding representation of instances.

[0064] S3 involves inputting the training data into various network models for feature extraction to obtain the features of the training instances.

[0065] Specifically, to obtain feature representations for downstream tasks, this invention utilizes commonly used models to extract features from training instances. In unsupervised visual representation learning, a commonly used feature extraction model, ResNet, is employed; in unsupervised sentence embedding representation learning, two common pre-trained models, BERT and RoBERTa, are used to obtain sentence representations; and in unsupervised graph representation learning, GIN is used as the basic skeleton to obtain graph representations.

[0066] S4 uses training instance features and loss functions to train multiple network models, and uses an optimizer to update model parameters to obtain multiple trained network models.

[0067] Specifically, similar to other contrastive learning models, this invention uses the InfoNCE loss function to train the model and the Adam optimizer for optimization.

[0068]

[0069] Among them, z i and z j It is a pair of positive samples, different data-augmented versions from the same training instance. k These are negative samples, derived from other training instances within the same batch size. Generally, K is set to K = 2(B-1), where B is the batch size. Sim(·) represents the cosine similarity, and τ is the temperature coefficient, which is an adjustable parameter.

[0070] In summary, experiments were conducted on three modalities (image, text, and graph) to verify the effectiveness of the novel mini-batch sampling proposed in this invention for improving negative sampling techniques in contrastive learning. Unlike previous methods such as DCL and HCL, which adjust the weights of negative samples, this invention utilizes neighborhood graphs to construct mini-batch data with hard samples, thereby improving the performance of contrastive learning.

[0071] Furthermore, experimental results show that the mini-batch sampling proposed in this invention can continuously improve the performance of the comparison methods. SimCLR achieves a relative improvement of 2.5% on ImageNet-100, SimCSE achieves a relative improvement of 1.4% on the standard STS task, and GraphCL achieves a relative improvement of 1.2% on the COLLAB dataset. This further demonstrates the effectiveness of the mini-batch sampling proposed in this invention.

[0072] To facilitate the widespread application of mini-batch sampling in contrastive learning, this invention also experimentally verified its efficiency. The experiments compared the time consumption of kNN and DkNN in constructing proximity graphs on two medium-sized datasets (STL10, ImageNet-100) and two large-scale datasets (Wikipedia, ImageNet). Experimental results show that the proposed mini-batch sampling method has good efficiency, achieving a 10-fold efficiency improvement compared to kNN on large-scale datasets.

[0073] To verify the effectiveness of DkNN, this invention conducts ablation studies by replacing the original strategy with different selection methods: (1) k-Nearest Neighbors (kNN) selects the k highest-scoring neighbors from the entire dataset. (2) Randomk-Nearest Neighbors (RkNN) randomly selects k instances as neighbors from the candidate set generated by kNN. (3) Debiased k-Nearest Neighbors (DkNN) is the strategy used in ProSampler. Experimental results show that DkNN outperforms other methods to some extent. Due to the false negative problem, both kNN and RkNN perform worse than the original contrastive learning method.

[0074] The method for improving contrastive learning through mini-batch sampling proposed in this embodiment of the invention utilizes a neighborhood graph to construct mini-batch data with hard samples, thereby improving the performance of contrastive learning.

[0075] To achieve the above embodiments, such as Figure 5 As shown, this embodiment also provides a device 10 for improving contrastive learning through mini-batch sampling. The device 10 includes a neighborhood graph construction module 100, a training data acquisition module 200, a feature extraction module 300, and a module optimization module 400.

[0076] The neighbor graph construction module 100 is used to obtain the instance dataset and construct the neighbor graph using the instance dataset.

[0077] The training data acquisition module 200 is used to sample a preset number of instances in the instance dataset on the neighborhood graph to obtain a preset number of training data; wherein, the training data is used to learn the embedding representation of the instances;

[0078] The feature extraction module 300 is used to input training data into various network models for feature extraction to obtain training instance features;

[0079] The module optimization module 400 is used to train multiple network models using training instance features and loss functions, and to update the model parameters using an optimizer to obtain multiple trained network models.

[0080] Furthermore, the aforementioned proximity graph construction module 100 is also used for:

[0081] A candidate set of instances is obtained by sampling multiple instances from the instance dataset using random sampling.

[0082] Query the instance candidate set for the instance's multiple nearest neighbor instance samples in the embedding space, and construct a proximity graph based on the multiple nearest neighbor instance samples.

[0083] Furthermore, the aforementioned training data acquisition module 200 is also used for:

[0084] Starting from an instance node in the neighborhood graph, the sampler iteratively returns to the starting point with a preset probability, or moves to a neighboring node at the current position with a probability proportional to the edge weight.

[0085] The process continues until a batch of data containing a preset number of instance nodes is collected, in order to obtain a preset number of training data.

[0086] Furthermore, the nearest neighbor graph is represented as follows:

[0087]

[0088] Where V = {v1, ..., vn} represents instances in the dataset. These are node pairs generated through similarity metrics, i.e., edges in the neighborhood graph.

[0089] The apparatus for improving contrastive learning through mini-batch sampling proposed in this embodiment of the invention utilizes a neighborhood graph to construct mini-batch data with hard samples, thereby improving the performance of contrastive learning.

[0090] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0091] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for improving contrastive learning through mini-batch sampling, characterized in that, Includes the following steps: Obtain the instance dataset, and use random sampling to sample multiple instances from the instance dataset to obtain an instance candidate set. In the instance candidate set, query the multiple nearest instance samples of the instance in the embedding space, and construct a neighborhood graph based on the multiple nearest instance samples. The nodes of the neighborhood graph are the instances in the instance dataset, and the edges of the neighborhood graph are constructed by learning the similarity between instances. Starting from an instance node in the neighborhood graph, a random walk sampler with restart is used to iteratively return to the starting point with a preset probability, or move to a neighboring node at the current position with a probability proportional to the edge weight, until a preset number of instance nodes are collected as training data; wherein, the training data consists of mini-batch data composed of similar nodes that are close in the embedding space, and the mini-batch data includes hard negative samples similar to the anchor instance, and the training data is used to learn the embedding representation of the instance; The training data is input into various network models for feature extraction to obtain the features of the training instances. Multiple network models are trained using training instance features and loss functions, and the optimized model parameters are updated to obtain multiple trained network models.

2. The method according to claim 1, characterized in that, The neighbor graph is represented as follows: Where V={v1,...,vn} represents instances in the dataset. These are node pairs generated through similarity metrics, i.e., edges in the neighborhood graph.

3. The method according to claim 1, characterized in that, The expression for constructing the proximity graph based on the plurality of nearest neighbor instance samples is: Where K is the instance sample, e m Let N be the candidate set of instances. vi Indicates instance v in the neighborhood graph i The neighbor of , · is the inner product operation.

4. The method according to claim 1, characterized in that, The expression for the loss function is: Among them, z i and z j It is a pair of positive samples, z k τ is the negative sample, B is the batch size, Sim(·) is the cosine similarity, and τ is the temperature coefficient.

5. An apparatus for improving contrastive learning through mini-batch sampling, characterized in that, include: The neighbor graph construction module is used to obtain the instance dataset. It samples multiple instances from the instance dataset using random sampling to obtain an instance candidate set. It queries the instance candidate set for the multiple nearest instance samples of each instance in the embedding space and constructs a neighbor graph based on the multiple nearest instance samples. The nodes of the neighbor graph are the instances in the instance dataset, and the edges of the neighbor graph are constructed by learning the similarity between instances. The training data acquisition module is used to start from an instance node on the neighborhood graph, and iteratively return to the starting point with a preset probability using a random walk sampler with restart, or move to the neighbor node of the current position with a probability proportional to the edge weight, until a preset number of instance nodes are collected as training data; wherein, the training data is a mini-batch of similar nodes that are close in the embedding space, and the mini-batch of data includes hard negative samples similar to the anchor instance, and the training data is used to learn the embedding representation of the instance; The feature extraction module is used to input training data into various network models for feature extraction to obtain the features of training instances; The module optimization module is used to train multiple network models using training instance features and loss functions, and then use the optimizer to update the model parameters to obtain multiple trained network models.

6. The apparatus according to claim 5, characterized in that, The proximity graph construction module is also used for: A candidate set of instances is obtained by sampling multiple instances from the instance dataset using random sampling. In the instance candidate set, query the multiple nearest instance samples of the instance in the embedding space, and construct a proximity graph based on the multiple nearest instance samples.

7. The apparatus according to claim 5, characterized in that, The training data acquisition module is also used for: Starting from an instance node on the neighborhood graph, the sampler iteratively returns to the starting point with a preset probability, or moves to a neighboring node at the current position with a probability proportional to the edge weight. The process continues until a batch of data containing a preset number of instance nodes is collected, in order to obtain a preset number of training data.

8. The apparatus according to claim 6, characterized in that, The neighbor graph is represented as follows: Where V={v1,...,vn} represents instances in the dataset. These are node pairs generated through similarity metrics, i.e., edges in the neighborhood graph.

Images