Self-supervised training method and apparatus

Abstract

Embodiments of the present application provide a self-supervised training method and device, relating to the field of artificial intelligence, which comprises: obtaining a picture sample dataset; performing a near neighbor sample search on the picture sample dataset to obtain a near neighbor sample set; performing near neighbor sample expansion based on the picture sample dataset and the near neighbor sample set to obtain expanded samples; performing high-level semantic aggregation of a training model based on the picture sample dataset, the near neighbor sample set and the expanded samples; and performing iterative training based on the high-level semantic aggregation result. The method provided by the embodiments of the present application can improve the expression ability of contrast learning, thereby improving the performance of the training task.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a self-supervised training method and apparatus. Background Technology

[0002] Supervised learning is a widely used pre-training method. However, because it relies on a large amount of manually labeled data, it is difficult to scale to large-scale datasets. For example, in practical applications, manually labeling data requires significant manpower. In addition, the generalization performance of existing vision tasks using supervised learning (such as ImageNet) as a pre-trained model is limited, especially in scenarios where the data distribution differs greatly from ImageNet.

[0003] Self-supervised learning, as an important direction for achieving general artificial intelligence, does not rely on data labeling at all. It mines the inherent structural information in large-scale unlabeled datasets through auxiliary tasks, learning transferable and highly generalizable feature representations. Effective self-supervised learning strategies can reduce the cost of manual labeling and avoid labeling errors. A good self-supervised pre-trained model can be adapted to various downstream tasks of classification and detection, achieving superior performance compared to fully supervised networks.

[0004] Self-supervised learning is an important component of building general network architectures. In the field of self-supervised learning, the network learns meaningful representations of input images by constructing pre-training tasks based on the content of the images themselves. For example, predicting the rotation angle of a rotated image, predicting the original color information of a grayscale image, predicting the original arrangement order of a randomly shuffled image, and so on.

[0005] Currently, contrastive learning, a self-supervised learning method based on instance recognition pre-training tasks, has made significant progress in the field of self-supervised learning. This learning paradigm has demonstrated its advantages over supervised learning in many downstream tasks. In contrastive learning, due to the lack of labels, each image and its other data-augmented version are treated as a separate category, and the network explicitly narrows the semantic representations between different data augmentations of the same sample; other images are treated as different categories, and the network explicitly widens the semantic representations between different samples. Under this setting, the model can learn representations invariant to cropping, color, and other transformations of the same image.

[0006] However, traditional contrastive learning only brings different augmented versions of the same sample closer together, while simultaneously pushing feature representations of different samples further apart. Therefore, invariant representations of semantically similar images are not explicitly learned. For large amounts of unlabeled data, designing effective pre-training tasks is a major challenge for self-supervised learning. Summary of the Invention

[0007] This application provides a self-supervised training method and apparatus to provide a way to perform self-supervised training on unlabeled data, thereby improving the expressive power of contrastive learning and thus improving the performance of the training task.

[0008] In a first aspect, embodiments of this application provide a self-supervised training method, including:

[0009] Obtain an image sample dataset; specifically, the image sample dataset can be a sample set containing multiple images. These images can be obtained through camera captures or downloaded from the internet; this application embodiment does not limit the source of the images.

[0010] A nearest neighbor search is performed on the image sample dataset to obtain a nearest neighbor sample set; specifically, this nearest neighbor sample set can be a collection of multiple nearest neighbor samples. These nearest neighbor samples can be samples from the aforementioned image sample dataset.

[0011] Nearest neighbor samples are used to augment the image sample dataset and the nearest neighbor sample set to obtain augmented samples; specifically, the augmented sample can be a synthetic image.

[0012] High-level semantic aggregation is performed on the training model based on image sample datasets, nearest neighbor sample sets, and augmented samples; specifically, this high-level semantic aggregation can be performed at a high level of the training model.

[0013] Iterative training is performed based on the high-level semantic aggregation results.

[0014] In this embodiment of the application, by performing nearest neighbor search on samples and clustering semantically similar samples, self-supervised training on unlabeled data can be achieved, which can improve the expressive power of contrastive learning and thus improve the performance of the training task.

[0015] To improve training efficiency, one possible implementation involves performing a nearest neighbor search on the image sample dataset to obtain a nearest neighbor set, including:

[0016] The image sample dataset is split into multiple sample data subsets;

[0017] Obtain any sample data from any subset of sample data and use it as the first training sample;

[0018] Perform a nearest neighbor search on the first training sample to obtain the nearest neighbor sample set.

[0019] To improve the efficiency of nearest neighbor search samples, one possible implementation involves performing nearest neighbor search on the first training sample, including:

[0020] Input all sample data from the subset of sample data into the training model to calculate and obtain the first feature of all sample data;

[0021] Calculate the cosine similarity between the first feature of the first training sample and the first feature of each other sample data;

[0022] Nearest neighbor sample search based on cosine similarity.

[0023] One possible implementation involves augmenting the image sample dataset with nearest neighbor samples, resulting in augmented samples including:

[0024] Get any nearest neighbor sample in the nearest neighbor sample set;

[0025] The first training sample is combined with the nearest neighbor sample to obtain the expanded sample.

[0026] In this embodiment of the application, by introducing expanded samples, the neighborhood space of positive samples can be expanded, thereby eliminating the performance gain or noise problem caused by the number of nearest neighbor samples.

[0027] One possible implementation involves training a high-level semantic aggregation model based on an image sample dataset, a nearest neighbor sample set, and augmented samples.

[0028] Obtain any extended nearest neighbor sample from the extended nearest neighbor sample set, wherein the extended nearest neighbor sample set includes the nearest neighbor sample set, the expanded sample, and the data augmentation sample, and the data augmentation sample is obtained by performing data augmentation on the first training sample;

[0029] The first training sample, the extended nearest neighbor sample, and the expanded sample are input into the first convolutional neural network to calculate the features of the first training sample, the features of the extended nearest neighbor sample, and the features of the expanded sample.

[0030] The first augmented training sample and the augmented extended nearest neighbor sample are input into the second convolutional neural network to calculate the features of the first augmented training sample and the features of the augmented extended nearest neighbor sample; wherein, the first augmented training sample is obtained by data augmentation of the first training sample, and the augmented extended nearest neighbor sample is obtained by data augmentation of the extended nearest neighbor sample;

[0031] High-level semantic aggregation is performed based on the features of the first training sample, the features of the extended nearest neighbor samples, the features of the expanded samples, the features of the first enhanced training sample, and the features of the enhanced extended nearest neighbor samples.

[0032] In this embodiment, features of samples are obtained through two neural networks, and semantic aggregation is performed based on the features of the two neural networks. This can effectively compare and learn the features of the two neural networks, thereby enabling more effective semantic aggregation and improving the accuracy of semantic aggregation.

[0033] One possible implementation includes:

[0034] The underlying semantic aggregation of the training model is based on image sample datasets, nearest neighbor sample sets, and augmented samples.

[0035] In this embodiment of the application, by performing semantic aggregation on the underlying layer of the training model, the semantic representation of the underlying layer can be enhanced, the model optimization speed can be accelerated, gradient inconsistency can be avoided, and the generalization performance of downstream tasks can be further improved.

[0036] To further enhance the underlying semantic representation and accelerate model optimization, one possible implementation involves training the model's underlying semantic aggregation based on image sample datasets, nearest neighbor sample sets, and augmented samples.

[0037] Semantic aggregation is performed on multiple underlying layers of the training model based on image sample datasets, nearest neighbor sample sets, and augmented samples.

[0038] In one possible implementation, the high-level semantic aggregation result includes a high-level loss, and iterative training based on the high-level semantic aggregation result includes:

[0039] The comprehensive loss is obtained based on the high-level loss and the low-level loss; the low-level loss is obtained by training the model by aggregating the low-level semantics based on the image sample dataset, the nearest neighbor sample set, and the expanded samples.

[0040] Iterative training is performed based on the comprehensive loss.

[0041] In this embodiment of the application, by designing high-level loss functions and low-level loss functions, high-level loss and low-level loss are obtained, and iterative training is performed based on high-level loss and low-level loss, which can improve the efficiency of self-supervised training.

[0042] One possible implementation method, which involves iterative training based on high-level semantic aggregation results, includes:

[0043] Detect the cumulative number of iterations during training;

[0044] If the cumulative number of iterations reaches a preset threshold, the network parameters of the trained model are recalculated to obtain the second feature of the sample data in the image sample dataset, and a nearest neighbor search is performed based on the second feature of the sample data in the image sample dataset.

[0045] In this embodiment, by recalculating the sample features based on the updated network parameters after a preset number of training rounds, the repeated calculation of features during each training round can be avoided, thereby improving the efficiency of nearest neighbor sample search and thus improving training efficiency.

[0046] Secondly, embodiments of this application provide a self-supervised training device, comprising:

[0047] The acquisition module is used to acquire image sample datasets;

[0048] The nearest neighbor search module is used to perform nearest neighbor search on the image sample dataset to obtain a set of nearest neighbor samples.

[0049] The sample augmentation module is used to augment neighbor samples based on the image sample dataset and the nearest neighbor sample set to obtain augmented samples.

[0050] The high-level semantic aggregation module is used for high-level semantic aggregation of the training model based on the image sample dataset, the nearest neighbor sample set, and the augmented samples.

[0051] The iterative training module is used for iterative training based on the high-level semantic aggregation results.

[0052] In one possible implementation, the nearest neighbor search module includes:

[0053] The splitting unit is used to split the image sample dataset into multiple subsets of sample data;

[0054] The selection unit is used to obtain any sample data from any subset of sample data as the first training sample;

[0055] The nearest neighbor search unit is used to search for nearest neighbor samples of the first training sample to obtain a set of nearest neighbor samples.

[0056] In one possible implementation, the aforementioned nearest neighbor search unit is also used to input all sample data in the sample data subset into the training model to calculate and obtain the first feature of all sample data.

[0057] Calculate the cosine similarity between the first feature of the first training sample and the first feature of each other sample data;

[0058] Nearest neighbor sample search based on cosine similarity.

[0059] In one possible implementation, the sample expansion module is also used to obtain any nearest neighbor sample in the nearest neighbor sample set;

[0060] The first training sample is combined with the nearest neighbor sample to obtain the expanded sample.

[0061] In one possible implementation, the aforementioned high-level semantic aggregation module includes:

[0062] The acquisition unit is used to acquire any extended nearest neighbor sample in the extended nearest neighbor sample set, wherein the extended nearest neighbor sample set includes the nearest neighbor sample set, the expanded sample and the data augmentation sample, and the data augmentation sample is obtained by data augmentation of the first training sample;

[0063] The first computing unit is used to input the first training sample, the extended nearest neighbor sample, and the expanded sample into the first convolutional neural network to calculate the features of the first training sample, the features of the extended nearest neighbor sample, and the features of the expanded sample.

[0064] The second computing unit is used to input the first augmented training sample and the augmented extended nearest neighbor sample into the second convolutional neural network to calculate and obtain the features of the first augmented training sample and the features of the augmented extended nearest neighbor sample; wherein, the first augmented training sample is obtained by data augmentation of the first training sample, and the augmented extended nearest neighbor sample is obtained by data augmentation of the extended nearest neighbor sample;

[0065] The high-level semantic aggregation unit is used to perform high-level semantic aggregation based on the features of the first training sample, the features of the extended nearest neighbor samples, the features of the expanded samples, the features of the enhanced first training sample, and the features of the enhanced extended nearest neighbor samples.

[0066] In one possible implementation, the above-mentioned device further includes:

[0067] The underlying semantic aggregation module is used for underlying semantic aggregation of the training model based on the image sample dataset, the nearest neighbor sample set, and the expanded samples.

[0068] In one possible implementation, the aforementioned underlying semantic aggregation module is also used to perform semantic aggregation on multiple underlying layers of the training model based on the image sample dataset, the nearest neighbor sample set, and the expanded samples.

[0069] In one possible implementation, the high-level semantic aggregation result includes the high-level loss, and the iterative training module is also used to obtain a comprehensive loss based on the high-level loss and the low-level loss; wherein, the low-level loss is obtained by training the model through low-level semantic aggregation based on the image sample dataset, the nearest neighbor sample set and the expanded samples.

[0070] Iterative training is performed based on the comprehensive loss.

[0071] In one possible implementation, the iterative training module is also used to detect the cumulative number of iterations during training;

[0072] If the cumulative number of iterations reaches a preset threshold, the network parameters of the trained model are recalculated to obtain the second feature of the sample data in the image sample dataset, and a nearest neighbor search is performed based on the second feature of the sample data in the image sample dataset.

[0073] Thirdly, embodiments of this application provide an electronic device, including:

[0074] A processor and memory, the memory for storing computer program code including instructions, which, when the processor reads instructions from the memory, cause the electronic device to perform the following steps:

[0075] Obtain the image sample dataset;

[0076] Perform a nearest neighbor search on the image sample dataset to obtain a nearest neighbor sample set;

[0077] Nearest neighbor samples are augmented based on the image sample dataset and the nearest neighbor sample set to obtain augmented samples;

[0078] High-level semantic aggregation of the training model based on image sample dataset, nearest neighbor sample set and augmented samples;

[0079] Iterative training is performed based on the high-level semantic aggregation results.

[0080] In one possible implementation, when the above instructions are executed by the electronic device, the electronic device performs a nearest neighbor search on the image sample dataset to obtain a nearest neighbor sample set, including the following steps:

[0081] The image sample dataset is split into multiple sample data subsets;

[0082] Obtain any sample data from any subset of sample data and use it as the first training sample;

[0083] Perform a nearest neighbor search on the first training sample to obtain the nearest neighbor sample set.

[0084] In one possible implementation, when the above instructions are executed by the electronic device, causing the electronic device to perform the step of searching for nearest neighbors of the first training sample, the steps include:

[0085] Input all sample data from the subset of sample data into the training model to calculate and obtain the first feature of all sample data;

[0086] Calculate the cosine similarity between the first feature of the first training sample and the first feature of each other sample data;

[0087] Nearest neighbor sample search based on cosine similarity.

[0088] In one possible implementation, when the above instructions are executed by the electronic device, the electronic device performs a step of augmenting neighbor samples based on the image sample dataset and the nearest neighbor sample set to obtain augmented samples, including:

[0089] Get any nearest neighbor sample in the nearest neighbor sample set;

[0090] The first training sample is combined with the nearest neighbor sample to obtain the expanded sample.

[0091] In one possible implementation, when the above instructions are executed by the electronic device, the electronic device performs the step of high-level semantic aggregation based on the image sample dataset, the nearest neighbor sample set, and the augmented samples to train the model, including:

[0092] Obtain any extended nearest neighbor sample from the extended nearest neighbor sample set, wherein the extended nearest neighbor sample set includes the nearest neighbor sample set, the expanded sample, and the data augmentation sample, and the data augmentation sample is obtained by performing data augmentation on the first training sample;

[0093] The first training sample, the extended nearest neighbor sample, and the expanded sample are input into the first convolutional neural network to calculate the features of the first training sample, the features of the extended nearest neighbor sample, and the features of the expanded sample.

[0094] The first augmented training sample and the augmented extended nearest neighbor sample are input into the second convolutional neural network to calculate the features of the first augmented training sample and the features of the augmented extended nearest neighbor sample; wherein, the first augmented training sample is obtained by data augmentation of the first training sample, and the augmented extended nearest neighbor sample is obtained by data augmentation of the extended nearest neighbor sample;

[0095] High-level semantic aggregation is performed based on the features of the first training sample, the features of the extended nearest neighbor samples, the features of the expanded samples, the features of the enhanced first training sample, and the features of the enhanced extended nearest neighbor samples.

[0096] In one possible implementation, when the above instructions are executed by the electronic device, the electronic device further performs the following steps:

[0097] The underlying semantic aggregation of the training model is based on image sample datasets, nearest neighbor sample sets, and augmented samples.

[0098] In one possible implementation, when the above instructions are executed by the electronic device, the electronic device performs the following steps to train the model based on the image sample dataset, the nearest neighbor sample set, and the augmented samples:

[0099] Semantic aggregation is performed on multiple underlying layers of the training model based on image sample datasets, nearest neighbor sample sets, and augmented samples.

[0100] In one possible implementation, the high-level semantic aggregation result includes a high-level loss. When the above instruction is executed by the electronic device, the electronic device performs the step of iterative training based on the high-level semantic aggregation result, including:

[0101] The comprehensive loss is obtained based on the high-level loss and the low-level loss; the low-level loss is obtained by training the model by aggregating the low-level semantics based on the image sample dataset, the nearest neighbor sample set, and the expanded samples.

[0102] Iterative training is performed based on the comprehensive loss.

[0103] In one possible implementation, when the above instructions are executed by the electronic device, causing the electronic device to perform the step of iterative training based on the high-level semantic aggregation results, the steps include:

[0104] Detect the cumulative number of iterations during training;

[0105] If the cumulative number of iterations reaches a preset threshold, the network parameters of the trained model are recalculated to obtain the second feature of the sample data in the image sample dataset, and a nearest neighbor search is performed based on the second feature of the sample data in the image sample dataset.

[0106] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method described in the first aspect.

[0107] Fifthly, embodiments of this application provide a computer program that, when executed by a computer, performs the method described in the first aspect.

[0108] In one possible design, the program in the fifth aspect can be stored wholly or partially on a storage medium packaged with the processor, or it can be stored wholly or partially on a memory not packaged with the processor. Attached Figure Description

[0109] Figure 1 A schematic diagram of an embodiment of the conventional training method provided in this application;

[0110] Figure 2 A schematic diagram of another embodiment of the conventional training method provided in this application;

[0111] Figure 3 This application provides a schematic diagram illustrating the effect of railway fault identification.

[0112] Figure 4a A schematic diagram illustrating the effect of remote sensing terrain segmentation provided in this application;

[0113] Figure 4b A schematic diagram illustrating the effect of medical image segmentation provided for this application;

[0114] Figure 5 This is a schematic diagram illustrating the effect of the product testing provided in this application;

[0115] Figure 6 A flowchart illustrating an embodiment of the self-supervised training method provided in this application;

[0116] Figure 7 This is a schematic diagram illustrating the synthesis effect of the expanded sample provided in the embodiments of this application;

[0117] Figure 8 A training model framework diagram provided for embodiments of this application;

[0118] Figure 9 This is a schematic diagram showing the performance comparison of the self-supervised training method provided in the embodiments of this application;

[0119] Figure 10 This is a schematic diagram of the structure of the self-supervised training device provided in the embodiments of this application;

[0120] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0121] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0122] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this application, unless otherwise stated, "a plurality of" means two or more.

[0123] Currently, traditional contrastive self-supervised learning includes the following two approaches:

[0124] Method 1

[0125] This approach is based on instance differentiation. Through a loss function design, it aims to narrow the distance between the feature representations B and B' of two data-augmented versions A and A' of the same image.

[0126]

[0127] To address the issue of non-smooth features between positive and negative samples, the above method employs a momentum moving average approach to encode B' and negative samples in the sample dataset. Figure 1 This is a flowchart illustrating Method 1. As can be seen from Method 1, it treats each sample and its different augmented versions as belonging to the same category, and all other samples as negative samples for a self-supervised pre-training task, without considering sample correlation. In the loss function design, it implicitly widens the feature representations of all other samples, failing to achieve effective semantic feature aggregation. Furthermore, traditional contrastive learning, by compressing the original input into a 128-dimensional feature vector and comparing these 128-dimensional feature vectors to achieve feature matching, loses most of the localization-sensitive semantic information.

[0128] Method 2

[0129] This approach is based on clustering. By combining clustering with deep learning, it learns useful general features. The process involves clustering these features, then using the clustering results as pseudo-labels to update the network parameters, and finally having the network predict these pseudo-labels. These two processes are performed sequentially. While the process seems simple, it achieves better performance than previous unsupervised methods. Figure 2 This is a flowchart illustrating method two.

[0130] The above process can also be expressed by the following formula;

[0131] The first formula is used to generate pseudo-labels y through clustering. pesudo :

[0132]

[0133] The second formula is used to calculate the loss value based on pseudo-labels, and then update the network parameters.

[0134]

[0135] Method 2 described above clusters features and then updates network parameters based on the clustering results as pseudo-labels. Since similarity calculation inevitably introduces noise, the features generated by the network, after clustering, tend to cluster around a specific cluster center, leaving other cluster centers without samples, thus limiting performance improvement.

[0136] Therefore, as can be seen from the above methods one and two, how to design effective pre-training tasks, gather semantically similar samples, and further improve the generalization performance of self-supervised training is an urgent problem to be solved.

[0137] Based on the above problems, this application proposes a self-supervised training method.

[0138] The self-supervised training method described above can be applied to the following scenarios:

[0139] Scenario 1: Image Classification

[0140] Image classification distinguishes different categories based on the semantic information of an image and outputs corresponding category labels. Specifically, a prime example is handwritten digit recognition: given a picture of a handwritten digit, the computer can determine which digit between 0 and 9 it represents. Image classification is a fundamental problem in computer vision and forms the basis for other high-level vision tasks such as image detection, image segmentation, object tracking, and behavior analysis. The core problem that image classification solves is predicting the category label for a given image. The challenges of image classification lie in the following aspects:

[0141] A. Multiple perspectives: Collecting images of the same object from multiple angles.

[0142] B. Multi-scale: In real life, the scale of many objects varies greatly.

[0143] C. Occlusion: The target object may be blocked, and sometimes only a small part of the object is visible.

[0144] D. Lighting conditions: At the pixel level, different intensities of light have a significant impact.

[0145] E. Intraclass differences: There are many different objects within the same class, each with its own shape.

[0146] F. Background interference: Complex backgrounds can affect object classification.

[0147] In addition, image classification can be divided into the following three categories:

[0148] Category 1: Cross-species classification

[0149] The aforementioned cross-species semantic-level image classification identifies different categories of objects at different species levels, with common examples including cat and dog classification. In such image classifications, because the categories belong to different species or broad classes, there is often significant inter-class variance, while intra-class variance is relatively small.

[0150] Category 2: Subclass Fine-grained Image Classification

[0151] Fine-grained image classification, compared to cross-species image classification, is at a lower level. It often involves classifying subclasses within the same broad category, such as different bird species, different dog breeds, or different car models. These tasks typically exhibit high within-class variance but low between-class variance. Therefore, training such a classifier requires it to recognize these regions, which is a more challenging problem than semantic-level cross-species image classification.

[0152] Category 3: Instance-level image classification

[0153] If we want to distinguish different individuals, rather than just species or subclasses, then it becomes a recognition problem, or instance-level image classification. The most typical task is face recognition. Face recognition tasks need to identify a person to complete tasks such as attendance tracking. Face recognition has always been a major topic in computer vision. Although it has undergone decades of development, it is still not completely solved. Its difficulties lie in classic problems such as occlusion, lighting, and large pose.

[0154] Figure 3 This is a schematic diagram for railway fault identification.

[0155] Scenario 2: Image Segmentation

[0156] The core of image segmentation is determining the category and instance of each pixel in an image. Specifically, image segmentation mainly consists of three tasks: semantic segmentation, instance segmentation, and panoptic segmentation. Semantic segmentation only determines the category information (including foreground and background) of each pixel; instance segmentation determines the foreground category to which each pixel belongs and which instance within that category; panoptic segmentation determines not only the foreground and background categories of each pixel but also the instance to which each pixel belongs within the foreground category. For example, in a street scene, semantic segmentation determines whether each pixel represents a road, car, building, or pedestrian. Instance segmentation determines whether each pixel is a car or a pedestrian, and which car or pedestrian it is; panoptic segmentation is a combination of the first two tasks. Its core issues lie in the following aspects:

[0157] A. Pixel-level segmentation requires guidance from high-level semantic information, but coarse-grained high-level semantic information has low resolution and is difficult to pinpoint precisely to each pixel.

[0158] B. Real-world images have complex backgrounds and foregrounds, resulting in very high levels of noise interference.

[0159] C. Different categories will occlude each other, and the distribution of different categories in the image is also uneven.

[0160] D. Small categories are difficult to define precisely, making it difficult to obtain semantic information.

[0161] E. Pixel features between classes may have high similarity, while the differences between different pixels within the same class may be large.

[0162] F. Different instances within the same class have very similar characteristics and semantic information.

[0163] Figure 4a and Figure 4b These are schematic diagrams of remote sensing terrain segmentation and medical image segmentation, respectively.

[0164] Scenario 3: Object Detection

[0165] The purpose of object detection is to determine whether there are target instances of a given category (such as cars, people, cats) in an image, and if so, to return the spatial location and category of each instance. Specifically, object detection can be divided into two sub-tasks: object classification and object localization. The object classification task is responsible for determining whether an object of interest (category) exists in the given image or a selected image region, outputting a series of labels with scores indicating the probability of the object's presence. The object localization task is responsible for locating the position and extent of objects of interest in the given image or a selected image region, outputting information such as the object's bounding box, center, or closed boundary. Generally, a rectangular bounding box is used to represent the object's position information in the image. Besides image classification, a core problem that object detection also needs to solve is the output of the spatial location of object instances, with the main challenges including the following aspects:

[0166] A. The object can appear anywhere in the image.

[0167] B. The objects are different sizes.

[0168] C. The object may be subject to interference factors such as deformation and occlusion.

[0169] If a rectangle is used to define an object, the rectangle can have different aspect ratios. Because objects have different aspect ratios, using the classic sliding window and image scaling scheme to solve the general object detection problem requires setting multiple sliding windows of different sizes and aspect ratios, which is too time-consuming and storage-intensive. Therefore, how to efficiently generate accurate object spatial position outputs has become one of the main concerns of subsequent algorithms.

[0170] Figure 5 This is a diagram illustrating the product inspection process.

[0171] Figure 6 A flowchart illustrating one embodiment of the self-supervised training method provided in this application includes:

[0172] Step 601: Obtain the sample dataset.

[0173] Specifically, the aforementioned sample dataset may include multiple samples. These samples may be unlabeled images; for example, one sample corresponds to one image, and the image does not contain any annotation information. These samples can be obtained by capturing images with a camera or downloaded from the internet. This application does not impose any special limitations on the source of the aforementioned samples. The aforementioned sample dataset can be used for training.

[0174] Step 602: Split the sample dataset to obtain multiple sample data subsets.

[0175] Optionally, before training on the aforementioned sample dataset, the sample dataset can be split into multiple sample data subsets. It is understood that the collection of these multiple sample data subsets constitutes the aforementioned sample dataset. The splitting of the data in the sample dataset can be random. For example, assuming the sample dataset has 1000 samples, these 1000 samples can be split into 10 groups, each containing 100 samples. That is, each group containing 100 samples is a sample data subset, where the 100 samples in each sample data subset are obtained through random splitting. It is understood that the above example only exemplifies the scenario of splitting into 10 sample data subsets and does not constitute a limitation on the embodiments of this application. In some embodiments, the aforementioned sample dataset can also be split into other numbers of sample data subsets. Furthermore, when splitting the aforementioned sample dataset, non-uniform splitting can also be performed; the embodiments of this application do not specifically limit the form of splitting.

[0176] Step 603: Obtain any one of the above multiple sample data subsets, and obtain any one sample from that sample data subset.

[0177] Specifically, after splitting the above sample dataset into multiple sample data subsets, one of these subsets can be arbitrarily selected as the current training subset (for ease of explanation, the "current training subset" will be referred to as the "first sample data subset"). Then, one sample can be arbitrarily selected from the first subset as the current training sample (for ease of explanation, the "current training sample" will be referred to as the "first training sample").

[0178] Step 604: Perform a nearest neighbor search on the first training sample to obtain a set of nearest neighbor samples.

[0179] Specifically, after determining the first training sample, a nearest neighbor search can be performed on it. In a concrete implementation, this nearest neighbor search can be performed by inputting the first sample data subset into a Convolutional Neural Network (CNN) for feature computation, thereby obtaining the feature value of each sample in the first sample data subset. It is understood that the CNN has initial network parameters, and the result of the feature computation is obtained based on these initial network parameters. For example, the CNN can be a standard ResNet50 network, and the feature value can be a 2048-dimensional feature vector obtained after average pooling in the last layer of the ResNet50 network.

[0180] Next, the cosine similarity between the first training sample and other training samples in the first sample data subset can be calculated. This cosine similarity can be obtained by calculating the feature values ​​of the first training sample and the feature values ​​of other training samples. For example, suppose the first sample data subset contains 100 samples, X1, X2…X100. X1 is the first training sample. Then, the feature value of X1 is Q1, and the feature values ​​of the remaining samples in the first sample data subset (e.g., X2-X100) are Q2, Q3…Q100. Next, the cosine similarity between Q1 and Q2, Q3…Q100 is calculated, resulting in 99 cosine similarities Y1, Y2…Y99. These 99 cosine similarities Y1, Y2…Y99 are then compared with a preset threshold, yielding K cosine similarities higher than the preset threshold. The samples corresponding to these K cosine similarities are the nearest neighbor sample set Z1, Z2…Zk. It is understood that the aforementioned preset threshold can be set in advance, and the embodiments of this application do not impose any special limitation on the value of the aforementioned preset threshold. It should be noted that the number K of samples in the aforementioned nearest neighbor sample set can also be set in advance. For example, the highest K cosine similarities can be selected from the aforementioned 99 cosine similarities Y1, Y2...Y99, and then the samples corresponding to the K cosine similarities can be obtained as the nearest neighbor sample set Z1, Z2...Zk.

[0181] Step 605: Augment the first training sample to obtain the first augmented training sample.

[0182] Specifically, selecting nearest neighbor samples by setting an appropriate K value in step 604 is very difficult. If the K value is small, it will bring samples that are already very similar in the feature space closer together, bringing limited performance gains. If the K value is large, it will bring in a lot of dissimilar noise samples, and bringing these samples closer together will break the local similarity constraint, which is not conducive to the learning of general representation features. To solve the above problems, the neighborhood space of positive samples (e.g., the positive sample can be the first training sample) can be expanded by data fusion. In specific implementation, the above data fusion can be carried out by data mixing (cutmix). For example, the above first training sample can be combined with any nearest neighbor sample to obtain the first expanded training sample. It is understood that the above nearest neighbor sample can be a nearest neighbor sample randomly selected from the above nearest neighbor sample set. The above data mixing method can be performed by the following formula:

[0183]

[0184] in, X1 is the first training sample, Xp is the nearest neighbor sample, and M is a mask with the same width and height as the first training sample.

[0185] For example, M∈{0,1} W×H Where W is the width of the first training sample and H is the height of the first training sample.

[0186] In other words, M can be used to determine which regions in the first training sample are replaced by neighboring samples.

[0187] Now combined Figure 7 The above data fusion will be explained. For example... Figure 7 As shown, image 700 is the first training sample, and image 710 is the nearest neighbor sample. Data fusion of images 700 and 710 yields image 720, which is the first augmented training sample. Image 720 includes a background region 7201 and a covering region 7202. The background region 7201 is the background of image 700, and the covering region 7202 is the region in image 710. The size and position of the covering region 7202 are determined by M.

[0188] By using the above-mentioned data mixing (cutmix) method, the training samples are synthesized with the nearest neighbor samples as augmentation samples, thereby reducing the influence of the K value in step 604.

[0189] Step 606: Perform data augmentation on the first training sample to obtain the first augmented training sample.

[0190] Specifically, the data augmentation method in step 606 can be to augment the color or brightness of the first training sample, thereby obtaining the first augmented training sample. In other words, the first augmented training sample is an enhanced version of the first training sample after the color or brightness has been augmented.

[0191] Step 607: Perform high-level semantic aggregation based on the first training sample, the nearest neighbor sample set, the first expanded training sample, and the first enhanced training sample.

[0192] Specifically, the aforementioned high layer can be the last layer in a convolutional neural network capable of semantic aggregation. Semantic aggregation can be achieved by designing a loss function to semantically aggregate positive samples, where positive samples can include the aforementioned first training sample, the nearest neighbor sample set, the first augmented training sample, and the first enhanced training sample.

[0193] Next, a random sample can be randomly selected from the extended nearest neighbor sample set and used as a positive sample of the first training sample. The extended nearest neighbor sample set can include the aforementioned nearest neighbor sample set, the first expanded training sample, and the first enhanced training sample. Then, the features of the first training sample, the features of the random sample, and the features of the first enhanced training sample can be brought closer together to design a loss function. In specific implementation, a loss function based on the features of the first training sample can be designed to bring the features of the first training sample and the features of the first enhanced training sample closer together. It is understood that the first feature of the first training sample can be the feature obtained after the first training sample has passed through a convolutional neural network (hereinafter referred to as the "first convolutional neural network"), and the feature of the first enhanced training sample can be the feature obtained after the first enhanced training sample has passed through another convolutional neural network (hereinafter referred to as the "second convolutional neural network"). The first enhanced training sample can be obtained by data augmentation of the first training sample. The data augmentation method can be referred to in step 606, and will not be repeated here. The network parameters of the second convolutional neural network can differ from those of the first convolutional neural network. This allows for semantic aggregation of the features processed by the convolutional neural network, i.e., feature comparison. For example, the loss function of the features of the first training sample can be represented by the following formula:

[0194]

[0195] Among them, L Q1Let Q1 be the loss of the features of the first training sample X1, Q1 be the features of the first training sample X1 after passing through the first convolutional neural network, Q1' be the features of the first augmented training sample X1' after passing through the second convolutional neural network, τ be a preset constant, and Qi' be the features of the negative sample Xi after passing through the second convolutional neural network. Here, the negative sample Xi is any sample in the sample data subset other than the first training sample X1, N is the total number of negative samples Xi in the sample data subset, and Q1' is the loss of the features of the first training sample X1 after passing through the first convolutional neural network. p To enhance the features of the random sample Xp after passing through the second convolutional neural network, the enhanced random sample Xp is obtained by performing data augmentation on the random sample Xp. The data augmentation method described above can be referred to in step 606, and will not be repeated here.

[0196] Similarly, a loss function for the features of the random samples can be designed to narrow the gap between the features of the random samples and the features of the enhanced random samples. The features of the random samples can be the features obtained after the random samples have passed through a first convolutional neural network, and the features of the enhanced random samples can be the features obtained after the random samples have passed through a second convolutional neural network. For example, the loss function for the features of the random samples can be represented by the following formula:

[0197]

[0198] Among them, L Qp Qp is the loss of the features of random sample Xp, Qp is the feature of random sample Xp after passing through the first convolutional neural network, and Qp' is the feature of enhanced random sample Xp' after passing through the second convolutional neural network.

[0199] Next, the aforementioned first extended training sample can be designed. The loss function is used to narrow down the features of the first expanded training sample to the features of the first augmented training sample and the augmented random sample. The features of the first expanded training sample are those obtained by processing the first expanded training sample through the first convolutional neural network. The features of the first augmented training sample are those obtained by processing the first augmented training sample through the second convolutional neural network; the features of the augmented random sample are those obtained by processing the augmented random sample through the second convolutional neural network. For example, the loss function for the features of the first expanded training sample can be represented by the following formula:

[0200]

[0201] in, For the first expanded training sample The loss of the features, where λ is a preset constant. For the first expanded training sample Its characteristics.

[0202] Therefore, the overall high-level loss function can be the loss L based on the features of the first training sample X1 mentioned above. Q1 Loss L of features of random sample Xp Qp and the first extended training sample Loss of features The overall loss function described above can be represented by the following formula:

[0203]

[0204] Among them, L High This represents the total loss for the top management.

[0205] Now combined Figure 8 The above-mentioned semantic aggregation method will be explained. Figure 8 This is a framework diagram for training the model. (Example) Figure 8 As shown, the first sample set 801 includes a first training sample 8011, a random sample 8012, and a first extended training sample 8013. The second sample set 802 includes a first enhanced training sample 8021 and an enhanced random sample 8022. The first enhanced training sample 8021 is obtained by data augmentation of the first training sample 8011; the enhanced random sample 8022 is obtained by data augmentation of the random sample 8012. Next, the first sample set 801 is input into the first convolutional neural network 811 for forward computation to obtain the first feature set 821. The first feature set 821 includes features 8211 of the first training sample 8011, features 8212 of the random sample 8012, and features 8213 of the first extended training sample 8013. The second sample set 802 can also be input into the second convolutional neural network 812 for forward computation to obtain the second feature set 822. The second feature set 822 includes features 8221 of the first enhanced training sample 8021 and features 8222 of the enhanced random sample 8022. Then, the first feature set 821 and the second feature set 822 are semantically aggregated. This allows for bringing training samples and their nearest neighbors closer together and expanding the training samples in the high-level feature space, while simultaneously pushing positive and negative samples further apart.

[0206] Step 608: Perform low-level semantic aggregation based on the first training sample, the nearest neighbor sample set, the first expanded training sample, and the first enhanced training sample.

[0207] Specifically, due to the lack of labeled sample data, contrastive learning is often difficult to optimize, especially the semantic representation of intermediate layers. Compared to fully supervised learning of intermediate layer representations, the representational power of features in unsupervised learning is significantly reduced. To prevent gradient inconsistencies and accelerate convergence and generalization, the contrastive loss can be extended to the intermediate layers of the network as another constraint on the learning process. Therefore, the above problems can also be avoided through low-level semantic aggregation. It is understandable that the aforementioned low-level semantic aggregation can also refer to the methods used for the aforementioned high-level semantic aggregation, for example, through methods such as... Figure 8 The training framework shown is designed with a loss function. For example, the loss function for feature Q1 of the first training sample X1 at layer j can be represented by the following formula:

[0208]

[0209] Among them, L Q1 j Let Q1 be the feature Q1 of the first training sample X1 in layer j, and Q1 be the loss. j Q1 represents the features of the first training sample X1 at layer j after passing through the first convolutional neural network. j’ Q represents the features of the first enhanced training sample X1' at layer j after passing through the second convolutional neural network. p j’ To enhance random sample X p 'Features at layer j after passing through the second convolutional neural network, Q' i j’ For negative sample X i The features at layer j after passing through the second convolutional neural network, where τ is a preset constant.

[0210] Next, the above-mentioned random sample X can be designed. p The features of the loss function at layer j. For example, the random sample X mentioned above... p Feature Q p The loss function at layer j can be characterized by the following formula:

[0211]

[0212] Among them, L Qp j For a random sample X p Feature Q p The loss at layer j, Q p j For a random sample X p Features at layer j after passing through the first convolutional neural network.

[0213] Next, the first extended training sample described above can be designed. The features of the loss function at layer j. For example, the random sample X mentioned above...p Feature Q p The loss function at layer j can be characterized by the following formula:

[0214]

[0215] in, For the first expanded training sample Features The loss at layer j For the first expanded training sample In the features of layer j, λ is a preset constant.

[0216] Therefore, the loss L at layer j can be based on the features Q1 of the first training sample X1 mentioned above. Q1 j Random sample X p Feature Q p Loss L at layer j Qp j and the first extended training sample Features Loss at layer j Obtain the total loss L of layer j. j For example, the total loss L of the above j-th layer. j It can be represented by the following formula:

[0217]

[0218] Step 609: Determine the total loss of the training model based on the loss of the high-level model and the loss of the low-level model.

[0219] Specifically, it can also be based on the loss of each layer in the aforementioned underlying layer (e.g., L). j Determine the total loss L at the bottom layer. Low For example, the total loss L at the bottom layer mentioned above Low It can be represented by the following formula:

[0220]

[0221] Where S is the total number of layers in the bottom layer.

[0222] When the total loss of the aforementioned high-level (e.g., L) is obtained High ) and the total loss at the bottom layer (e.g., L) Low After that, the total loss L of the entire training model can be determined based on the total loss of the higher-level and lower-level models mentioned above. total For example, L total =L High +L Low It is understandable that the above-mentioned training model can be as follows: Figure 8 The training framework shown.

[0223] Step 610: Iteratively train the training model based on the high-level total loss and the low-level total loss.

[0224] Specifically, after obtaining the aforementioned total loss at the higher levels and total loss at the lower levels, one can base the analysis on the aforementioned total loss at the higher levels (e.g., L). High The training model is iteratively trained.

[0225] Alternatively, the loss can be based on the combined loss of the total loss at the higher level and the total loss at the lower level (e.g., L). total The training model is iteratively trained.

[0226] Furthermore, to improve the training speed of the aforementioned training model, after every N epochs of training, the features of the samples can be recalculated based on the network parameters of the current training model to obtain a new set of nearest neighbor samples. That is, before the completion of the aforementioned N epochs of training, the features of each sample are not recalculated in each epoch. Here, N is a preset number, and N is a positive integer. This application embodiment does not impose a special limitation on the number of N. It is understood that the aforementioned epoch can be a training round, which can be training a subset of sample data. The training process for each sample in this subset can be performed through steps 603-609. When all samples in a subset of sample data have been trained, the training of that subset of sample data is complete, which is equivalent to completing that round of training (one epoch). After completing N epochs of training, for example, after completing the training of N subsets of sample data, the sample features can be recalculated based on the updated network parameters of the training model, thereby obtaining a new set of nearest neighbor samples. It is understood that the network parameters of the above-mentioned training model can be the parameters of the convolutional neural network (e.g., ResNet50 network) in step 604. The convolutional neural network in step 604 can be any one of the above-mentioned first convolutional neural network or second convolutional neural network, or it can be a convolutional neural network other than the above-mentioned first convolutional neural network and / or second convolutional neural network, such as a third convolutional neural network. The embodiments of this application do not impose any special limitations on the convolutional neural network in step 604.

[0227] It is understandable that when iteratively training the model, a condition for ending the training can be preset. For example, this condition could be that the loss of the training model reaches a certain threshold, or it could be that the number of training iterations reaches a certain threshold. This application does not specifically limit the above-mentioned conditions for ending the training. Therefore, after each round of training, it can be determined whether the current training meets the ending condition. If the current training meets the ending condition, the current training can be terminated; if the current training does not meet the ending condition, the next round of training can continue.

[0228] It should be noted that in each epoch of training, after the nearest neighbor sample set is calculated using the first training sample, since this nearest neighbor sample set is obtained based on the similarity between the features of the first training sample and the features of other samples, it means that the features of all samples in that epoch of training have been obtained and can be stored. Therefore, to improve efficiency, in the same epoch of training, when training samples other than the first training sample, even if the network parameters of the convolutional neural network (e.g., ResNet50 network) are updated, the features of all samples in that epoch of training are not recalculated. In other words, the features of all the samples already stored can be used to calculate similarity, thereby obtaining the nearest neighbor sample set of the current sample, which improves the calculation speed of the nearest neighbor sample set.

[0229] Furthermore, within the same epoch of training, the network parameters of the convolutional neural network in step 604 may not need to be updated. If the network parameters of the trained model are not updated, features can be calculated for all samples based on the current network parameters, and a set of nearest neighbor samples can be obtained based on the calculated sample features.

[0230] After N epochs of training, the network parameters can be updated. At this point, in the next epoch of training, the features of all samples can be calculated based on the updated network parameters, and the nearest neighbor sample set can be recalculated based on the newly calculated features. This results in a new nearest neighbor sample set, which allows for more accurate acquisition of nearest neighbor samples by adjusting the network parameters, thereby improving the accuracy of semantic clustering.

[0231] Figure 9 This is a diagram illustrating the performance comparison of self-supervised training methods. For example... Figure 9As shown, the training method provided in this application (hereinafter referred to as HSA for ease of explanation) can significantly improve the expressive power of the original contrastive learning. On several widely evaluated unsupervised benchmarks, the overall performance of the proposed self-supervised pre-trained features is compared. Using ResNet-50 as the base model, the training method provided in this application can significantly improve the performance of various tasks. Furthermore, it surpasses the performance of fully supervised expression in various metrics such as few-shot classification, detection, and segmentation.

[0232] Table 1 shows the top-1 accuracy of linear classification evaluation.

[0233]

[0234] As shown in Table 1, the training method (HSA) of this application embodiment can achieve performance comparable to that of fully supervised training, outperform MoCo v2 by 5.3%, and outperform SwAV by 1.1%.

[0235] Table 2 shows the accuracy of the few-sample learning task.

[0236]

[0237]

[0238] As shown in Table 2, the training method (HSA) of this application embodiment achieves a top-1 classification accuracy of 75.1% using only 10% of the labeled data, which is close to the performance of fully supervised learning (76.5%).

[0239] In this embodiment, by performing nearest neighbor search on samples and clustering semantically similar samples, self-supervised training on unlabeled data can be achieved, which can improve the expressive power of contrastive learning and thus improve the performance of the training task.

[0240] It is understood that in the above embodiments, steps 601-610 are optional steps. This application only provides a feasible embodiment, and may also include more or fewer steps than steps 601-610. This application does not limit this.

[0241] Figure 10 This is a schematic diagram of the structure of one embodiment of the self-supervised training device of this application, as shown below. Figure 10 As shown, the self-supervised training device 1000 may include: an acquisition module 1010, a nearest neighbor search module 1020, a sample augmentation module 1030, a high-level semantic aggregation module 1040, and an iterative training module 1050; wherein,

[0242] Module 1010 is used to acquire image sample datasets;

[0243] The nearest neighbor search module 1020 is used to perform nearest neighbor sample search on the image sample dataset to obtain a nearest neighbor sample set.

[0244] The sample augmentation module 1030 is used to augment neighbor samples based on the image sample dataset and the nearest neighbor sample set to obtain augmented samples.

[0245] The high-level semantic aggregation module 1040 is used for high-level semantic aggregation of the training model based on the image sample dataset, the nearest neighbor sample set and the augmented samples.

[0246] The iterative training module 1050 is used for iterative training based on the high-level semantic aggregation results.

[0247] In one possible implementation, the nearest neighbor search module 1020 includes:

[0248] Splitting unit 1021 is used to split the image sample dataset into multiple sample data subsets;

[0249] Select unit 1022 is used to obtain any sample data in any subset of sample data as the first training sample;

[0250] The nearest neighbor search unit 1023 is used to search for nearest neighbor samples of the first training sample to obtain a set of nearest neighbor samples.

[0251] In one possible implementation, the nearest neighbor search unit 1023 is also used to input all sample data in the sample data subset into the training model to calculate and obtain the first feature of all sample data.

[0252] Calculate the cosine similarity between the first feature of the first training sample and the first feature of each other sample data;

[0253] Nearest neighbor sample search based on cosine similarity.

[0254] In one possible implementation, the sample expansion module 1030 is also used to obtain any nearest neighbor sample in the nearest neighbor sample set;

[0255] The first training sample is combined with the nearest neighbor sample to obtain the expanded sample.

[0256] In one possible implementation, the aforementioned high-level semantic aggregation module 1040 includes:

[0257] The acquisition unit 1041 is used to acquire any extended nearest neighbor sample in the extended nearest neighbor sample set, wherein the extended nearest neighbor sample set includes the nearest neighbor sample set, the expanded sample and the data augmentation sample, and the data augmentation sample is obtained by data augmentation of the first training sample;

[0258] The first computing unit 1042 is used to input the first training sample, the extended nearest neighbor sample and the expanded sample into the first convolutional neural network to calculate the features of the first training sample, the features of the extended nearest neighbor sample and the features of the expanded sample.

[0259] The second computing unit 1043 is used to input the first enhanced training sample and the enhanced extended nearest neighbor sample into the second convolutional neural network to calculate and obtain the features of the first enhanced training sample and the features of the enhanced extended nearest neighbor sample; wherein, the first enhanced training sample is obtained by data augmentation of the first training sample, and the enhanced extended nearest neighbor sample is obtained by data augmentation of the extended nearest neighbor sample.

[0260] The high-level semantic aggregation unit 1044 is used for high-level semantic aggregation, and is used to perform high-level semantic aggregation based on the features of the first training sample, the features of the extended nearest neighbor sample, the features of the expanded sample, the features of the enhanced first training sample, and the features of the enhanced extended nearest neighbor sample.

[0261] In one possible implementation, the device 1000 further includes:

[0262] The underlying semantic aggregation module 1060 is used for underlying semantic aggregation of the training model based on the image sample dataset, the nearest neighbor sample set, and the expanded samples.

[0263] In one possible implementation, the aforementioned underlying semantic aggregation module 1060 is also used to perform semantic aggregation on multiple underlying layers of the training model based on the image sample dataset, the nearest neighbor sample set, and the expanded samples.

[0264] In one possible implementation, the high-level semantic aggregation result includes the high-level loss, and the iterative training module 1050 is also used to obtain a comprehensive loss based on the high-level loss and the low-level loss; wherein, the low-level loss is obtained by training the model through low-level semantic aggregation based on the image sample dataset, the nearest neighbor sample set and the expanded samples.

[0265] Iterative training is performed based on the comprehensive loss.

[0266] In one possible implementation, the iterative training module 1050 is also used to detect the cumulative number of iterations of the training.

[0267] If the cumulative number of iterations reaches a preset threshold, the network parameters of the trained model are recalculated to obtain the second feature of the sample data in the image sample dataset, and a nearest neighbor search is performed based on the second feature of the sample data in the image sample dataset.

[0268] Figure 10 The self-supervised training apparatus provided in the illustrated embodiment can be used to execute this application. Figures 1-9The implementation principle and technical effects of the method embodiment shown can be further referred to the relevant description in the method embodiment.

[0269] The above should be understood Figure 10 The division of modules in the self-supervised training device shown is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. These modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. For example, the detection module can be a separate processing element or integrated into a chip in an electronic device. The implementation of other modules is similar. Furthermore, these modules can be fully or partially integrated together, or implemented independently. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0270] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, these modules can be integrated together as a System-On-a-Chip (SOC).

[0271] Figure 11 This is a schematic diagram of the structure of an embodiment of the electronic device of this application; as shown Figure 11 As shown, the aforementioned electronic device can be a data processing device, such as a computer, or a circuit device built into the aforementioned data processing device. This electronic device can be used to execute this application. Figures 1-9 The functions / steps in the method provided in the illustrated embodiment.

[0272] like Figure 11 As shown, the electronic device 1100 is presented in the form of a general-purpose computing device.

[0273] The aforementioned electronic device 1100 may include: one or more processors 1110; a communication interface 1120; a memory 1130; a communication bus 1140 connecting different system components (including the memory 1130 and the processor 1110); a database 1150; and one or more computer programs.

[0274] The aforementioned one or more computer programs are stored in the aforementioned memory, and the aforementioned one or more computer programs include instructions that, when executed by the aforementioned electronic device, cause the aforementioned electronic device to perform the following steps:

[0275] Obtain the image sample dataset;

[0276] Perform a nearest neighbor search on the image sample dataset to obtain a nearest neighbor sample set;

[0277] Nearest neighbor samples are augmented based on the image sample dataset and the nearest neighbor sample set to obtain augmented samples;

[0278] High-level semantic aggregation of the training model based on image sample dataset, nearest neighbor sample set and augmented samples;

[0279] Iterative training is performed based on the high-level semantic aggregation results.

[0280] In one possible implementation, when the above instructions are executed by the electronic device, the electronic device performs a nearest neighbor search on the image sample dataset to obtain a nearest neighbor sample set, including the following steps:

[0281] The image sample dataset is split into multiple sample data subsets;

[0282] Obtain any sample data from any subset of sample data and use it as the first training sample;

[0283] Perform a nearest neighbor search on the first training sample to obtain the nearest neighbor sample set.

[0284] In one possible implementation, when the above instructions are executed by the electronic device, causing the electronic device to perform the step of searching for nearest neighbors of the first training sample, the steps include:

[0285] Input all sample data from the subset of sample data into the training model to calculate and obtain the first feature of all sample data;

[0286] Calculate the cosine similarity between the first feature of the first training sample and the first feature of each other sample data;

[0287] Nearest neighbor sample search based on cosine similarity.

[0288] In one possible implementation, when the above instructions are executed by the electronic device, the electronic device performs a step of augmenting neighbor samples based on the image sample dataset and the nearest neighbor sample set to obtain augmented samples, including:

[0289] Get any nearest neighbor sample in the nearest neighbor sample set;

[0290] The first training sample is combined with the nearest neighbor sample to obtain the expanded sample.

[0291] In one possible implementation, when the above instructions are executed by the electronic device, the electronic device performs the step of high-level semantic aggregation based on the image sample dataset, the nearest neighbor sample set, and the augmented samples to train the model, including:

[0292] Obtain any extended nearest neighbor sample from the extended nearest neighbor sample set, wherein the extended nearest neighbor sample set includes the nearest neighbor sample set, the expanded sample, and the data augmentation sample, and the data augmentation sample is obtained by performing data augmentation on the first training sample;

[0293] The first training sample, the extended nearest neighbor sample, and the expanded sample are input into the first convolutional neural network to calculate the features of the first training sample, the features of the extended nearest neighbor sample, and the features of the expanded sample.

[0294] The first augmented training sample and the augmented extended nearest neighbor sample are input into the second convolutional neural network to calculate the features of the first augmented training sample and the features of the augmented extended nearest neighbor sample; wherein, the first augmented training sample is obtained by data augmentation of the first training sample, and the augmented extended nearest neighbor sample is obtained by data augmentation of the extended nearest neighbor sample;

[0295] High-level semantic aggregation is performed based on the features of the first training sample, the features of the extended nearest neighbor samples, the features of the expanded samples, the features of the enhanced first training sample, and the features of the enhanced extended nearest neighbor samples.

[0296] In one possible implementation, when the above instructions are executed by the electronic device, the electronic device further performs the following steps:

[0297] The underlying semantic aggregation of the training model is based on image sample datasets, nearest neighbor sample sets, and augmented samples.

[0298] In one possible implementation, when the above instructions are executed by the electronic device, the electronic device performs the following steps to train the model based on the image sample dataset, the nearest neighbor sample set, and the augmented samples:

[0299] Semantic aggregation is performed on multiple underlying layers of the training model based on image sample datasets, nearest neighbor sample sets, and augmented samples.

[0300] In one possible implementation, the high-level semantic aggregation result includes a high-level loss. When the above instruction is executed by the electronic device, the electronic device performs the step of iterative training based on the high-level semantic aggregation result, including:

[0301] The comprehensive loss is obtained based on the high-level loss and the low-level loss; the low-level loss is obtained by training the model by aggregating the low-level semantics based on the image sample dataset, the nearest neighbor sample set, and the expanded samples.

[0302] Iterative training is performed based on the comprehensive loss.

[0303] In one possible implementation, when the above instructions are executed by the electronic device, causing the electronic device to perform the step of iterative training based on the high-level semantic aggregation results, the steps include:

[0304] Detect the cumulative number of iterations during training;

[0305] If the cumulative number of iterations reaches a preset threshold, the network parameters of the trained model are recalculated to obtain the second feature of the sample data in the image sample dataset, and a nearest neighbor search is performed based on the second feature of the sample data in the image sample dataset.

[0306] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 1100. In other embodiments of this application, the electronic device 1100 may also adopt different interface connection methods or a combination of multiple interface connection methods as described in the above embodiments.

[0307] It is understood that the aforementioned electronic devices, etc., include hardware structures and / or software modules corresponding to the execution of each function in order to achieve the above-mentioned functions. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0308] This application embodiment can divide the above-mentioned electronic device into functional modules according to the method example described above. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0309] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0310] In the embodiments of this application, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0311] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0312] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A self-supervised training method, characterized in that, The method includes: Obtain the image sample dataset; A nearest neighbor search is performed on the image sample dataset to obtain a nearest neighbor sample set; Based on the image sample dataset and the nearest neighbor sample set, nearest neighbor samples are expanded to obtain expanded samples; High-level semantic aggregation of the training model is performed based on the image sample dataset, the nearest neighbor sample set, and the augmented samples; Iterative training is performed based on the results of high-level semantic aggregation; The process of augmenting neighbor samples based on the image sample dataset and the nearest neighbor sample set to obtain augmented samples includes: Obtain any nearest neighbor sample from the nearest neighbor sample set; The first training sample is combined with the nearest neighbor sample to obtain an augmented sample. The first training sample is the sample data in the image sample dataset. The high-level semantic aggregation result includes a high-level loss, and the iterative training based on the high-level semantic aggregation result includes: A comprehensive loss is obtained based on the high-level loss and the low-level loss; wherein, the low-level loss is obtained by training the model by performing low-level semantic aggregation based on the image sample dataset, the nearest neighbor sample set, and the augmented samples; Iterative training is performed based on the aforementioned comprehensive loss.

2. The method according to claim 1, characterized in that, The process of performing a nearest neighbor search on the image sample dataset to obtain a nearest neighbor sample set includes: The image sample dataset is split into multiple sample data subsets; Obtain any sample data from any subset of the sample data and use it as the first training sample; Perform a nearest neighbor search on the first training sample to obtain a nearest neighbor sample set.

3. The method according to claim 2, characterized in that, The nearest neighbor search of the first training sample includes: The first feature of all sample data is obtained by inputting all sample data in the subset of sample data into the training model. Calculate the cosine similarity between the first feature of the first training sample and the first feature of each other sample data; Nearest neighbor sample search is performed based on the cosine similarity.

4. The method according to claim 1, characterized in that, The high-level semantic aggregation of the model trained based on the image sample dataset, the nearest neighbor sample set, and the augmented samples includes: Obtain any extended nearest neighbor sample from the extended nearest neighbor sample set, wherein the extended nearest neighbor sample set includes the nearest neighbor sample set, the expanded sample, and the data augmentation sample, and the data augmentation sample is obtained by performing data augmentation on the first training sample; The first training sample, the extended nearest neighbor sample, and the expanded sample are input into the first convolutional neural network to calculate the features of the first training sample, the features of the extended nearest neighbor sample, and the features of the expanded sample. The first enhanced training sample and the enhanced extended nearest neighbor sample are input into the second convolutional neural network for calculation to obtain the features of the first enhanced training sample and the features of the enhanced extended nearest neighbor sample; wherein, the first enhanced training sample is obtained by data augmentation of the first training sample, and the enhanced extended nearest neighbor sample is obtained by data augmentation of the extended nearest neighbor sample; High-level semantic aggregation is performed based on the features of the first training sample, the features of the extended nearest neighbor sample, the features of the expanded sample, the features of the first enhanced training sample, and the features of the enhanced extended nearest neighbor sample.

5. The method according to claim 1, characterized in that, The loss at the bottom layer is obtained by semantically aggregating multiple bottom layers of the trained model based on the image sample dataset, the nearest neighbor sample set, and the augmented samples.

6. The method according to claim 1, characterized in that, The iterative training based on high-level semantic aggregation results includes: Detect the cumulative number of iterations during training; If the cumulative number of iterations in the iterative training reaches a preset cumulative number of iterations threshold, the network parameters of the trained model are recalculated to obtain the second feature of the sample data in the image sample dataset, and a nearest neighbor sample search is performed based on the second feature of the sample data in the image sample dataset.

7. A self-supervised training device, characterized in that, include: The acquisition module is used to acquire image sample datasets; The nearest neighbor search module is used to perform nearest neighbor sample search on the image sample dataset to obtain a nearest neighbor sample set; The sample augmentation module is used to augment neighbor samples based on the image sample dataset and the nearest neighbor sample set to obtain augmented samples. The high-level semantic aggregation module is used to perform high-level semantic aggregation of the training model based on the image sample dataset, the nearest neighbor sample set, and the augmented samples. An iterative training module is used to perform iterative training based on the high-level semantic aggregation results, which include high-level losses. The sample expansion module is specifically used to obtain any nearest neighbor sample in the nearest neighbor sample set; The first training sample is combined with the nearest neighbor sample to obtain an augmented sample. The first training sample is the sample data in the image sample dataset. The iterative training module is specifically used to obtain a comprehensive loss based on the high-level loss and the low-level loss; wherein, the low-level loss is obtained by training the model by performing low-level semantic aggregation based on the image sample dataset, the nearest neighbor sample set, and the expanded samples; Iterative training is performed based on the aforementioned comprehensive loss.

8. The apparatus according to claim 7, characterized in that, The nearest neighbor search module includes: The splitting unit is used to split the image sample dataset into multiple sample data subsets; The selection unit is used to acquire any sample data from any subset of the sample data as the first training sample. The nearest neighbor search unit is used to perform a nearest neighbor search on the first training sample to obtain a nearest neighbor sample set.

9. The apparatus according to claim 8, characterized in that, The nearest neighbor search unit is also used to input all sample data in the sample data subset into the training model to calculate and obtain the first feature of all sample data. Calculate the cosine similarity between the first feature of the first training sample and the first feature of each other sample data; Nearest neighbor sample search is performed based on the cosine similarity.

10. The apparatus according to claim 7, characterized in that, The high-level semantic aggregation module includes: An acquisition unit is used to acquire any extended nearest neighbor sample in the extended nearest neighbor sample set, wherein the extended nearest neighbor sample set includes the nearest neighbor sample set, the expanded sample, and the data augmentation sample, and the data augmentation sample is obtained by performing data augmentation on the first training sample; The first computing unit is used to input the first training sample, the extended nearest neighbor sample, and the expanded sample into the first convolutional neural network to calculate the features of the first training sample, the features of the extended nearest neighbor sample, and the features of the expanded sample. The second computing unit is used to input the first enhanced training sample and the enhanced extended nearest neighbor sample into the second convolutional neural network to calculate and obtain the features of the first enhanced training sample and the features of the enhanced extended nearest neighbor sample; wherein, the first enhanced training sample is obtained by data augmentation of the first training sample, and the enhanced extended nearest neighbor sample is obtained by data augmentation of the extended nearest neighbor sample. A high-level semantic aggregation unit is used to perform high-level semantic aggregation based on the features of the first training sample, the features of the extended nearest neighbor sample, the features of the expanded sample, the features of the first enhanced training sample, and the features of the enhanced extended nearest neighbor sample.

11. The apparatus according to claim 7, characterized in that, The loss at the bottom layer is obtained by semantically aggregating multiple bottom layers of the trained model based on the image sample dataset, the nearest neighbor sample set, and the augmented samples.

12. The apparatus according to claim 7, characterized in that, The iterative training module is also used to detect the cumulative number of iterations in the training; If the cumulative number of iterations in the iterative training reaches a preset cumulative number of iterations threshold, the network parameters of the trained model are recalculated to obtain the second feature of the sample data in the image sample dataset, and a nearest neighbor sample search is performed based on the second feature of the sample data in the image sample dataset.

Images