Image feature extraction network training method and related device
By performing category annotation and momentum comparison learning on the image dataset, the problem of insufficient robustness of the image feature extraction network in high-similar scenarios is solved, and the image feature quality and processing effect are improved.
Patent Information
- Application Number
- CN202510738894.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-04
- Publication Date
- 2025-08-22
AI Technical Summary
When the existing image feature extraction network is trained in high-similar scenarios, it is easy to construct different images affiliated with the same object into positive and negative samples at the same time, resulting in poor robustness and poor quality of the extracted image features, which affects the subsequent image processing effect.
By classifying the image dataset, query samples and positive samples are constructed while avoiding the image of the same object as negative samples, and using momentum comparison learning to train the image feature extraction network, priority is given to retaining difficult samples in the training batch, and update using negative sample queues to improve the network's feature expression ability.
It improves the robustness of the image feature extraction network, improves the image feature quality in high-similar scenes, and ensures subsequent image processing effects.
Smart Images
Figure CN120526258A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of neural network technology, and in particular to an image feature extraction network training method and related devices. Background Art
[0002] In certain application scenarios involving images, it is necessary to obtain image features for the images in the scene, and then perform certain operations based on the obtained image features. It is understandable that the acquisition of image features is a relatively basic and critical operation, and the quality of the acquired image features will directly affect the subsequent operation results. For example, in the textbook retrieval scenario, it is necessary to obtain image features for the target textbook image, and then search for images similar to the target textbook image in the textbook image library based on the acquired image features. If the acquired image features can better represent the textbook, then a better retrieval result can be obtained, otherwise a better retrieval result cannot be obtained.
[0003] A common way to obtain image features is to input the image into an image feature extraction network and use the feature vectors of the network's intermediate hidden layer to represent the image. Currently, image feature extraction networks are mostly trained using contrastive learning. When constructing query samples, positive samples, and negative samples, they are randomly extracted from the full image dataset. For some image processing scenarios, such as document image processing, textbook image processing, and medical image processing, the images in their image datasets have high image similarity. Using the current sample construction method, different images belonging to the same object are easily constructed as positive and negative samples at the same time, affecting the effectiveness of contrastive learning. This results in poor robustness of the trained image feature extraction network, poor quality of the extracted image features, and inability to guarantee the effectiveness of subsequent image processing.
[0004] Therefore, how to provide an image feature extraction network training method suitable for scenes with high image similarity to improve the robustness of the image feature extraction network, thereby improving the feature quality of images in scenes with high image similarity and ensuring the subsequent image processing effect has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the Invention
[0005] In view of the above problems, this application provides an image feature extraction network training method and related devices to achieve image feature extraction network training suitable for scenes with high image similarity, thereby improving the robustness of the image feature extraction network, thereby improving the feature quality of images in scenes with high image similarity, and ensuring the purpose of subsequent image processing effects. The specific solution is as follows:
[0006] The first aspect of the present application provides an image feature extraction network training method, comprising:
[0007] Label each image in the image dataset according to its category. Images with the same category label belong to the same object, and images with different categories belong to different objects.
[0008] Constructing query samples and positive samples based on images of a first annotated category, and constructing negative samples based on images of a second annotated category, where the first annotated category is different from the second annotated category;
[0009] An image feature extraction network is trained using the query sample, the positive sample, and the negative sample.
[0010] In a possible implementation, constructing a query sample and a positive sample based on the image of the first labeled category includes:
[0011] Selecting a first image and a second image from the images of the first labeled category;
[0012] performing enhancement processing on the first image and the second image respectively to obtain a first enhanced image and a second enhanced image;
[0013] The query sample and the positive sample are determined based on the first enhanced image and the second enhanced image.
[0014] In a possible implementation, constructing negative samples based on images of the second labeled category includes:
[0015] Selecting a third image from the images of the second labeled category;
[0016] The third image is enhanced to obtain a third enhanced image, where the third image is the negative sample.
[0017] In a possible implementation, the training of an image feature extraction network using the query sample, the positive sample, and the negative sample includes:
[0018] For each training batch, the query sample, the positive sample and the negative sample are used to train the image feature extraction network based on the negative sample queue using momentum contrast learning, and the negative sample queue is updated based on the negative samples in the training batch. The head of the updated negative sample queue retains the difficult samples among the negative samples in the training batch.
[0019] In a possible implementation, updating the negative sample queue based on the negative samples in the training batch includes:
[0020] Calculating the similarity between each negative sample in the training batch and the query sample;
[0021] Determine a sample category corresponding to each negative sample based on the similarity between each negative sample in the training batch and the query sample, a high similarity threshold and a low similarity threshold corresponding to the training batch, wherein the sample category includes difficult samples, common samples, and easy samples;
[0022] Negative samples whose sample category is the easy sample are discarded, negative samples whose sample category is the difficult sample are inserted into the head of the negative sample queue, and negative samples whose sample category is the ordinary sample are inserted behind the difficult samples in the negative sample queue in chronological order, and the queue length of the negative sample queue is maintained at no more than the queue length threshold.
[0023] In a possible implementation, calculating the similarity between each negative sample in the training batch and the query sample includes:
[0024] Obtaining features of each negative sample in the training batch and features of the query sample;
[0025] Normalizing the features of each negative sample in the training batch to obtain normalized negative sample features;
[0026] Normalizing the query sample features to obtain normalized query sample features;
[0027] For each negative sample in the batch, the similarity between the negative sample and the query sample is calculated based on the normalized negative sample features and the normalized query sample features.
[0028] In one possible implementation, the method of using momentum contrastive learning to train an image feature extraction network based on a negative sample queue includes:
[0029] Inputting the query sample into the main encoder to obtain query sample features;
[0030] Inputting the positive sample into a momentum encoder to obtain positive sample features;
[0031] Obtaining negative sample features in historical training batches from the negative sample queue;
[0032] A preset contrastive learning loss function is used to calculate contrastive loss values between the query sample features, the positive sample features, and the negative sample features, and parameters in the main encoder and the momentum encoder are adjusted according to the contrastive loss values.
[0033] In a possible implementation, calculating the contrast loss value between the query sample feature, the positive sample feature, and the negative sample feature using a preset contrastive learning loss function includes:
[0034] Based on a first contrastive learning loss function, calculating a positive sample similarity between the query sample feature and the positive sample feature, and calculating a negative sample similarity between the query sample feature and each of the negative sample features;
[0035] A comparison loss value between the query sample feature, the positive sample feature representation, and the negative sample feature representation is determined according to the positive sample similarity and each of the negative sample similarities.
[0036] In a possible implementation, calculating the contrast loss value between the query sample feature, the positive sample feature, and the negative sample feature using a preset contrastive learning loss function includes:
[0037] Based on the second contrastive learning loss function and the preset weights corresponding to each sample category, calculating the positive sample similarity between the query sample feature and the positive sample feature, and calculating the negative sample weighted similarity between the query sample feature and each negative sample feature;
[0038] The comparison loss values among the query sample feature, the positive sample feature, and the negative sample feature are determined according to the positive sample similarity and the weighted similarities of each of the negative samples.
[0039] In a possible implementation, after calculating the contrastive loss values between the query sample feature, the positive sample feature, and the negative sample feature using a preset contrastive learning loss function, the method further includes:
[0040] Calculating a contrast loss change rate based on the contrast loss value and the contrast loss value corresponding to the previous training batch;
[0041] The high similarity threshold and the low similarity threshold corresponding to the training batch are adjusted based on the contrast loss change rate to obtain the high similarity threshold and the low similarity threshold corresponding to the next training batch.
[0042] A second aspect of the present application provides an image feature extraction network training device, comprising:
[0043] The category labeling unit is used to label the categories of each image in the image dataset. Images with the same category labeling belong to the same object, and images with different categories belong to different objects.
[0044] a sample construction unit, configured to construct a query sample and a positive sample based on an image of a first annotated category, and to construct a negative sample based on an image of a second annotated category, wherein the first annotated category is different from the second annotated category;
[0045] A training unit is used to train an image feature extraction network using the query sample, the positive sample and the negative sample.
[0046] In a possible implementation, the sample construction unit includes a first sample construction unit;
[0047] The first sample construction unit is specifically used to:
[0048] Selecting a first image and a second image from the images of the first labeled category;
[0049] performing enhancement processing on the first image and the second image respectively to obtain a first enhanced image and a second enhanced image;
[0050] The query sample and the positive sample are determined based on the first enhanced image and the second enhanced image.
[0051] In a possible implementation, the sample construction unit includes a second sample construction unit;
[0052] The second sample construction unit is specifically used to:
[0053] Selecting a third image from the images of the second labeled category;
[0054] The third image is enhanced to obtain a third enhanced image, where the third image is the negative sample.
[0055] In a possible implementation, the training unit is specifically configured to:
[0056] For each training batch, the query sample, the positive sample and the negative sample are used to train the image feature extraction network based on the negative sample queue using momentum contrast learning, and the negative sample queue is updated based on the negative samples in the training batch. The head of the updated negative sample queue retains the difficult samples among the negative samples in the training batch.
[0057] The training unit includes a negative sample queue updating unit, and the negative sample queue updating unit includes:
[0058] A similarity calculation unit, configured to calculate the similarity between each negative sample in the training batch and the query sample;
[0059] a sample category determination unit, configured to determine a sample category corresponding to each negative sample based on the similarity between each negative sample in the training batch and the query sample, a high similarity threshold and a low similarity threshold corresponding to the training batch, wherein the sample categories include difficult samples, common samples, and easy samples;
[0060] An updating unit is used to discard negative samples whose sample category is the easy sample, insert negative samples whose sample category is the difficult sample into the head of the negative sample queue, insert negative samples whose sample category is the common sample behind the difficult sample in the negative sample queue in chronological order, and maintain the queue length of the negative sample queue not exceeding the queue length threshold.
[0061] In a possible implementation, the similarity calculation unit is specifically configured to:
[0062] Obtaining features of each negative sample in the training batch and features of the query sample;
[0063] Normalizing the features of each negative sample in the training batch to obtain normalized negative sample features;
[0064] Normalizing the query sample features to obtain normalized query sample features;
[0065] For each negative sample in the batch, the similarity between the negative sample and the query sample is calculated based on the normalized negative sample features and the normalized query sample features.
[0066] In a possible implementation, the training unit includes:
[0067] A sample feature determination unit is configured to input the query sample into a main encoder to obtain query sample features; input the positive sample into a momentum encoder to obtain positive sample features; and obtain negative sample features in a historical training batch from the negative sample queue;
[0068] A contrastive learning unit is used to calculate the contrastive loss value between the query sample feature, the positive sample feature and the negative sample feature using a preset contrastive learning loss function, and adjust the parameters in the main encoder and the momentum encoder according to the contrastive loss value.
[0069] In a possible implementation, the contrastive learning unit includes: a first contrastive loss value calculation unit;
[0070] The first contrast loss value calculation unit is specifically used to calculate the positive sample similarity between the query sample feature and the positive sample feature based on the first contrast learning loss function, and to calculate the negative sample similarity between the query sample feature and each of the negative sample features; determine the contrast loss value between the query sample feature, the positive sample feature representation and each of the negative sample feature representations according to the positive sample similarity and each of the negative sample similarities.
[0071] In a possible implementation, the contrastive learning unit includes: a second contrastive loss value calculation unit;
[0072] The second contrast loss value calculation unit is specifically used to calculate the positive sample similarity between the query sample feature and the positive sample feature, and calculate the negative sample weighted similarity between the query sample feature and each negative sample feature based on the second contrast learning loss function and the preset weights corresponding to each sample category;
[0073] The comparison loss values among the query sample feature, the positive sample feature, and the negative sample feature are determined according to the positive sample similarity and the weighted similarities of each of the negative samples.
[0074] In a possible implementation, the apparatus further includes: a threshold adjustment unit;
[0075] The threshold adjustment unit is used to calculate the contrast loss value between the query sample feature, the positive sample feature and the negative sample feature using the preset contrast learning loss function, and then calculate the contrast loss change rate based on the contrast loss value and the contrast loss value corresponding to the previous training batch; adjust the high similarity threshold and the low similarity threshold corresponding to the training batch based on the contrast loss change rate to obtain the high similarity threshold and the low similarity threshold corresponding to the next training batch.
[0076] A third aspect of the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the image feature extraction network training method of the first aspect or any implementation of the first aspect.
[0077] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:
[0078] The memory is used to store computer programs;
[0079] The processor is used to execute the computer program so that the electronic device can implement the image feature extraction network training method of the above-mentioned first aspect or any implementation method of the first aspect.
[0080] In a fifth aspect, the present application provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can use the image feature extraction network training method of the above-mentioned first aspect or any implementation of the first aspect.
[0081] By means of the above technical solution, the image feature extraction network training method and related devices provided by the present application first perform category labeling on each image in the image data set, where images labeled with the same category belong to the same object, and images of different categories belong to different objects; then, query samples and positive samples are constructed based on images of the first labeled category, and negative samples are constructed based on images of a second labeled category different from the first labeled category; this construction method can avoid the problem of simultaneously constructing different images belonging to the same object as positive samples and negative samples, and therefore, the image feature extraction network is trained using the above query samples, positive samples and negative samples by adopting a comparative learning method, which can improve the feature expression ability of the image feature extraction network and make the trained image feature extraction network more robust. By using the image feature extraction network to extract image features of scenes with high image similarity, the quality of image features can be improved and the subsequent image processing effect can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS
[0082] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.
[0083] Figure 1 A flowchart of a network training method for image feature extraction provided in an embodiment of the present application;
[0084] Figure 2 A schematic diagram of the structure of an image feature extraction network training device provided in an embodiment of the present application;
[0085] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0086] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.
[0087] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0088] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0089] In certain application scenarios involving images, it is necessary to obtain image features for the images in the scene, and then perform certain operations based on the obtained image features. It is understandable that the acquisition of image features is a relatively basic and critical operation, and the quality of the acquired image features will directly affect the subsequent operation results. For example, in the textbook retrieval scenario, it is necessary to obtain image features for the target textbook image, and then search for images similar to the target textbook image in the textbook image library based on the acquired image features. If the acquired image features can better represent the textbook, then a better retrieval result can be obtained, otherwise a better retrieval result cannot be obtained.
[0090] A common way to obtain image features is to input the image into an image feature extraction network and use the feature vectors of the network's intermediate hidden layer to represent the image. Currently, image feature extraction networks are mostly trained using contrastive learning. When constructing query samples, positive samples, and negative samples, they are randomly extracted from the full image dataset. For some image processing scenarios, such as document image processing, textbook image processing, and medical image processing, the images in their image datasets have high image similarity. Using the current sample construction method, different images belonging to the same object are easily constructed as positive and negative samples at the same time, affecting the effectiveness of contrastive learning. This results in poor robustness of the trained image feature extraction network, poor quality of the extracted image features, and inability to guarantee the effectiveness of subsequent image processing.
[0091] In order to solve the above problems, the present invention provides an image feature extraction network training method. The image feature extraction network training method of the present invention is described in detail below with reference to the accompanying drawings.
[0092] Reference Figure 1 , Figure 1 A flowchart of an image feature extraction network training method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, an image feature extraction network training method provided in an embodiment of the present application may include the following steps, which are described in detail below.
[0093] S101: Label each image in the image dataset according to its category. Images labeled with the same category belong to the same object, and images labeled with different categories belong to different objects.
[0094] For ease of understanding, assume that the image dataset is an image dataset corresponding to a textbook retrieval scenario. The image dataset corresponds to 100 textbooks, each textbook has 100 pages, and the image dataset contains images of each page taken at different angles and distances. In this application, each page can be regarded as a category, and a textbook corresponds to 100 categories. The image dataset corresponds to 10,000 categories. For each category, the image corresponding to the category belongs to the same page of the same textbook. For different categories, the images corresponding to different categories belong to different pages of the same textbook, or different pages of different textbooks.
[0095] In the present application, the method of classifying each image in the image data set may be manual labeling or automatic labeling, and the present application does not impose any limitation on this.
[0096] S102: constructing a query sample and a positive sample based on an image of a first annotated category, and constructing a negative sample based on an image of a second annotated category, where the first annotated category is different from the second annotated category;
[0097] In the present application, positive sample pairs and negative samples are constructed based on this method, which can ensure that different images belonging to the same object are only constructed as positive samples and not as negative samples.
[0098] S103: Using the query sample, the positive sample, and the negative sample, train an image feature extraction network.
[0099] The image feature extraction network training method provided in this embodiment first labels the categories of each image in the image data set, and images labeled with the same category belong to the same object, and images of different categories belong to different objects; then, query samples and positive samples are constructed based on images of the first labeled category, and negative samples are constructed based on images of a second labeled category different from the first labeled category; this construction method can avoid the problem of constructing different images belonging to the same object as positive samples and negative samples at the same time, and therefore, the image feature extraction network is trained using the above-mentioned query samples, positive samples and negative samples by adopting a comparative learning method, which can improve the feature expression ability of the image feature extraction network and make the trained image feature extraction network more robust. The image feature extraction network is used to extract image features of scenes with high image similarity, which can improve the quality of image features and thus ensure the subsequent image processing effect.
[0100] In a possible implementation, constructing a query sample and a positive sample based on the image of the first labeled category includes:
[0101] S201: Selecting a first image and a second image from images of the first labeled category;
[0102] In this application, two images may be randomly selected from the images of the first annotated category as the first image and the second image, or two images may be selected from the images of the first annotated category according to a certain rule as the first image and the second image, and this application does not impose any limitation on this.
[0103] S202: Performing enhancement processing on the first image and the second image respectively to obtain a first enhanced image and a second enhanced image;
[0104] The purpose of image enhancement processing is to improve the diversity of images. After all, when constructing an image dataset, collecting images under various conditions (such as different lighting, different clarity, etc.) will greatly consume manpower. Therefore, in this application, in order to reduce the pressure of image acquisition, only images under some conditions that cannot be simulated by image enhancement (such as different shooting angles, different shooting distances, etc.) are manually collected. For some conditions that are easy to simulate by image enhancement, manual collection will no longer be relied upon. Therefore, the images in the image dataset in this application cannot cover images under various conditions. Therefore, in this application, after selecting two images from images of the same labeled category, the selected images need to be enhanced separately to obtain two enhanced images, and the positive sample pairs are constructed based on the two enhanced images. This processing method can increase the diversity of positive sample pairs and improve the training effect.
[0105] S203: Determine the query sample and the positive sample based on the first enhanced image and the second enhanced image.
[0106] In the present application, any one of the first enhanced image and the second enhanced image can be determined as the query sample, and the other one can be determined as the positive sample. For example, the first enhanced image can be determined as the query sample, and the second enhanced image can be determined as the positive sample; or, the first enhanced image can be determined as the positive sample, and the second enhanced image can be determined as the query sample.
[0107] In a possible implementation, constructing negative samples based on images of the second labeled category includes:
[0108] Selecting a third image from the images of the second labeled category;
[0109] The third image is enhanced to obtain a third enhanced image, where the third image is the negative sample.
[0110] In this application, referring to the description of the specific implementation of constructing query samples and positive samples above, based on the same requirements, it is also necessary to enhance the images selected from the second annotated category to obtain enhanced images, and construct these enhanced images as negative samples. This processing method can increase the diversity of negative samples and improve training results. It will not be further described here.
[0111] In a possible implementation, the training of an image feature extraction network using the query sample, the positive sample, and the negative sample includes:
[0112] For each training batch, the query sample, the positive sample and the negative sample are used to train the image feature extraction network based on the negative sample queue using momentum contrast learning, and the negative sample queue is updated based on the negative samples in the training batch. The head of the updated negative sample queue retains the difficult samples among the negative samples in the training batch.
[0113] In this application, as a possible implementation, when updating the negative sample queue based on the negative samples in the training batch, the update can be based on the similarity between each negative sample and the query sample. In one possible implementation, the difficult sample is a negative sample in the training batch whose similarity with the query sample meets a preset condition. The preset condition can be greater than a preset similarity threshold, which is not limited in this application.
[0114] In this application, in a possible implementation, the negative sample queue includes negative sample features and corresponding sample category labels.
[0115] In this application, since momentum contrastive learning is used to train the image feature extraction network based on a negative sample queue, the image feature extraction network needs to include a main encoder and a momentum encoder. Since this structure is the basic structure of momentum contrastive learning, it will not be described in detail here.
[0116] In this embodiment, when the image feature extraction network is trained using momentum contrastive learning, the difficult samples in each training batch are preferentially retained, so that the difficult samples in the image data set are fully trained, thereby further improving the contrastive learning effect and the robustness of the image feature extraction network, and further ensuring the quality of the extracted image features and the subsequent image processing effect.
[0117] In a possible implementation, updating the negative sample queue based on the negative samples in the training batch includes:
[0118] S301: Calculating the similarity between each negative sample in the training batch and the query sample;
[0119] S302: Determine a sample category corresponding to each negative sample based on the similarity between each negative sample in the training batch and the query sample, a high similarity threshold and a low similarity threshold corresponding to the training batch, wherein the sample category includes difficult samples, common samples, and easy samples;
[0120] In the present application, a high similarity threshold (θ_high) and a low similarity threshold (θ_low) can be set. The high similarity thresholds corresponding to different training batches can be the same or different. Similarly, the low similarity thresholds corresponding to different training batches can be the same or different. If the similarity between the negative sample and the query sample is greater than the high similarity threshold, the sample category corresponding to the negative sample is determined to be a difficult sample. If the similarity between the negative sample and the query sample is less than the low similarity threshold, the sample category corresponding to the negative sample is determined to be an easy sample. If the similarity between the negative sample and the query sample is between the high similarity threshold and the low similarity threshold, the sample category corresponding to the negative sample is determined to be a normal sample.
[0121] In the present application, since the negative sample queue stores negative sample features, in the present application, the similarity between the negative sample and the query sample may be the similarity between the negative sample features and the query sample features.
[0122] S303: Discard the negative samples whose sample category is the easy samples, insert the negative samples whose sample category is the difficult samples into the head of the negative sample queue, insert the negative samples whose sample category is the ordinary samples into the negative sample queue behind the difficult samples in chronological order, and maintain the queue length of the negative sample queue not exceeding the queue length threshold.
[0123] It should be noted that after the negative sample with the sample category of the difficult sample is inserted into the head of the negative sample queue, and the negative sample with the sample category of the normal sample is inserted behind the difficult sample in the negative sample queue in chronological order, if the queue length exceeds the queue length threshold, the B samples at the tail of the negative sample queue are eliminated, where B is used to indicate the batch size.
[0124] It should also be noted that the negative samples inserted into the negative sample queue need to record their corresponding sample category identifiers.
[0125] In a possible implementation, calculating the similarity between each negative sample in the training batch and the query sample includes:
[0126] S401: Obtaining features of each negative sample in the training batch and features of the query sample;
[0127] In this application, assuming that the number of negative samples corresponding to a training batch is B, the features of each negative sample in the training batch can be represented as fi = f1, f2, ..., fB, where i = 1, 2, ..., B, and B is the number of negative samples corresponding to a batch. It should be noted that the features of each negative sample in the training batch can be generated by the momentum encoder, and the query sample features can be generated by the main encoder. The specific generation method will not be described in detail.
[0128] S402: Normalizing the features of each negative sample in the training batch to obtain normalized negative sample features;
[0129] In a possible implementation, the features of each negative sample in the training batch can be L2 normalized to obtain the normalized negative sample features. .
[0130] S403: performing normalization processing on the query sample features to obtain normalized query sample features;
[0131] Assuming that the query sample feature is q, in one possible implementation, the query sample feature q can be L2 normalized to obtain the normalized query sample feature .
[0132] S404: For each negative sample in the batch, calculate the similarity between the negative sample and the query sample based on the normalized negative sample features and the normalized query sample features.
[0133] In one possible implementation, the cosine similarity between the two can be calculated:
[0134] .
[0135] In one possible implementation, the method of using momentum contrastive learning to train an image feature extraction network based on a negative sample queue includes:
[0136] Inputting the query sample into the main encoder to obtain query sample features;
[0137] Inputting the positive sample into a momentum encoder to obtain positive sample features;
[0138] Obtaining negative sample features in historical training batches from the negative sample queue;
[0139] A preset contrastive learning loss function is used to calculate contrastive loss values between the query sample features, the positive sample features, and the negative sample features, and parameters in the main encoder and the momentum encoder are adjusted according to the contrastive loss values.
[0140] Since the difficult samples in the negative samples in the historical training batches are preferentially retained in the negative sample queue, when the negative sample features in the historical training batches are obtained from the negative sample queue, the difficult samples in the negative samples in the historical training batches can be trained again, making their training more sufficient.
[0141] In this application, the contrastive learning loss function can be a general InfoNCE loss function. In one possible implementation, the InfoNCE loss function can be minimized based on gradient descent to maximize the similarity between the query sample features and the positive sample features, while minimizing the similarity between the query sample features and the negative sample features. Training ends when the InfoNCE loss function value stabilizes.
[0142] In a possible implementation, calculating the contrast loss value between the query sample feature, the positive sample feature, and the negative sample feature using a preset contrastive learning loss function includes:
[0143] Based on a first contrastive learning loss function, calculating a positive sample similarity between the query sample feature and the positive sample feature, and calculating a negative sample similarity between the query sample feature and each of the negative sample features;
[0144] A comparison loss value between the query sample feature, the positive sample feature representation, and the negative sample feature representation is determined according to the positive sample similarity and each of the negative sample similarities.
[0145] In a possible implementation, calculating the contrast loss value between the query sample feature, the positive sample feature, and the negative sample feature using a preset contrastive learning loss function includes:
[0146] Based on the second contrastive learning loss function and the preset weights corresponding to each sample category, calculating the positive sample similarity between the query sample feature and the positive sample feature, and calculating the negative sample weighted similarity between the query sample feature and each negative sample feature;
[0147] The comparison loss values among the query sample feature, the positive sample feature, and the negative sample feature are determined according to the positive sample similarity and the weighted similarities of each of the negative samples.
[0148] In one possible implementation, the target sample category corresponding to the negative sample feature and the target weight corresponding to the target sample category are determined, where the target sample category is any one of a difficult sample, a normal sample, and an easy sample; then, for each negative sample, the negative sample similarity between the query sample feature and the negative sample feature is calculated, and the product of the target weight and the negative sample similarity is calculated as the negative sample weighted similarity. In one possible implementation, the target sample category corresponding to the negative sample feature can be determined based on the sample category identifier corresponding to the negative sample recorded in the negative sample queue, and the weight corresponding to each sample category can be preset, for example, the weight corresponding to the difficult sample is 1.5, the weight corresponding to the normal sample is 1, and the weight corresponding to the easy sample is 0.5.
[0149] In a possible implementation, after calculating the contrastive loss values between the query sample feature, the positive sample feature, and the negative sample feature using a preset contrastive learning loss function, the method further includes:
[0150] Calculating a contrast loss change rate based on the contrast loss value and the contrast loss value corresponding to the previous training batch;
[0151] The high similarity threshold and the low similarity threshold corresponding to the training batch are adjusted based on the contrast loss change rate to obtain the high similarity threshold and the low similarity threshold corresponding to the next training batch.
[0152] In this application, the formula θ_high can be used 新 =θ_high 旧 -α·ΔL and θ_low 新 =θ_low 旧 +β·ΔL, where ΔL is the rate of change of contrast loss, α and β are adjustment coefficients, which are set according to task experience (e.g. α=0.01, β=0.005), θ_high 旧 and θ_low 旧 are the high similarity threshold and low similarity threshold corresponding to the training batch, θ_high 新 and θ_low 新 are the high similarity threshold and low similarity threshold corresponding to the next training batch respectively. This adjustment method can gradually tighten the high similarity threshold in the later stages of training and gradually relax the low similarity threshold in the later stages of training.
[0153] The above describes an image feature extraction network training method provided by an embodiment of the present application. The following describes a device for executing the above image feature extraction network training method.
[0154] See also Figure 2 , Figure 2This is a schematic diagram of the structure of an image feature extraction network training device provided in an embodiment of the present application. Figure 2 As shown, the image feature extraction network training device includes:
[0155] A category labeling unit 11 is used to label each image in the image data set according to its category. Images with the same category labeling belong to the same object, and images with different categories belong to different objects.
[0156] A sample construction unit 12 is configured to construct a query sample and a positive sample based on an image of a first annotated category, and to construct a negative sample based on an image of a second annotated category, wherein the first annotated category is different from the second annotated category;
[0157] The training unit 13 is configured to train an image feature extraction network using the query sample, the positive sample, and the negative sample.
[0158] In a possible implementation, the sample construction unit includes a first sample construction unit;
[0159] The first sample construction unit is specifically used to:
[0160] Selecting a first image and a second image from the images of the first labeled category;
[0161] performing enhancement processing on the first image and the second image respectively to obtain a first enhanced image and a second enhanced image;
[0162] The query sample and the positive sample are determined based on the first enhanced image and the second enhanced image.
[0163] In a possible implementation, the sample construction unit includes a second sample construction unit;
[0164] The second sample construction unit is specifically used to:
[0165] Selecting a third image from the images of the second labeled category;
[0166] The third image is enhanced to obtain a third enhanced image, where the third image is the negative sample.
[0167] In a possible implementation, the training unit is specifically configured to:
[0168] For each training batch, the query sample, the positive sample and the negative sample are used to train the image feature extraction network based on the negative sample queue using momentum contrast learning, and the negative sample queue is updated based on the negative samples in the training batch. The head of the updated negative sample queue retains the difficult samples among the negative samples in the training batch.
[0169] The training unit includes a negative sample queue updating unit, and the negative sample queue updating unit includes:
[0170] A similarity calculation unit, configured to calculate the similarity between each negative sample in the training batch and the query sample;
[0171] a sample category determination unit, configured to determine a sample category corresponding to each negative sample based on the similarity between each negative sample in the training batch and the query sample, a high similarity threshold and a low similarity threshold corresponding to the training batch, wherein the sample categories include difficult samples, common samples, and easy samples;
[0172] An updating unit is used to discard negative samples whose sample category is the easy sample, insert negative samples whose sample category is the difficult sample into the head of the negative sample queue, insert negative samples whose sample category is the common sample behind the difficult sample in the negative sample queue in chronological order, and maintain the queue length of the negative sample queue not exceeding the queue length threshold.
[0173] In a possible implementation, the similarity calculation unit is specifically configured to:
[0174] Obtaining features of each negative sample in the training batch and features of the query sample;
[0175] Normalizing the features of each negative sample in the training batch to obtain normalized negative sample features;
[0176] Normalizing the query sample features to obtain normalized query sample features;
[0177] For each negative sample in the batch, the similarity between the negative sample and the query sample is calculated based on the normalized negative sample features and the normalized query sample features.
[0178] In a possible implementation, the training unit includes:
[0179] A sample feature determination unit is configured to input the query sample into a main encoder to obtain query sample features; input the positive sample into a momentum encoder to obtain positive sample features; and obtain negative sample features in a historical training batch from the negative sample queue;
[0180] A contrastive learning unit is used to calculate the contrastive loss value between the query sample feature, the positive sample feature and the negative sample feature using a preset contrastive learning loss function, and adjust the parameters in the main encoder and the momentum encoder according to the contrastive loss value.
[0181] In a possible implementation, the contrastive learning unit includes: a first contrastive loss value calculation unit;
[0182] The first contrast loss value calculation unit is specifically used to calculate the positive sample similarity between the query sample feature and the positive sample feature based on the first contrast learning loss function, and to calculate the negative sample similarity between the query sample feature and each of the negative sample features; determine the contrast loss value between the query sample feature, the positive sample feature representation and each of the negative sample feature representations according to the positive sample similarity and each of the negative sample similarities.
[0183] In a possible implementation, the contrastive learning unit includes: a second contrastive loss value calculation unit;
[0184] The second contrast loss value calculation unit is specifically used to calculate the positive sample similarity between the query sample feature and the positive sample feature, and calculate the negative sample weighted similarity between the query sample feature and each negative sample feature based on the second contrast learning loss function and the preset weights corresponding to each sample category;
[0185] The comparison loss values among the query sample feature, the positive sample feature, and the negative sample feature are determined according to the positive sample similarity and the weighted similarities of each of the negative samples.
[0186] In a possible implementation, the apparatus further includes: a threshold adjustment unit;
[0187] The threshold adjustment unit is used to calculate the contrast loss value between the query sample feature, the positive sample feature and the negative sample feature using the preset contrast learning loss function, and then calculate the contrast loss change rate based on the contrast loss value and the contrast loss value corresponding to the previous training batch; adjust the high similarity threshold and the low similarity threshold corresponding to the training batch based on the contrast loss change rate to obtain the high similarity threshold and the low similarity threshold corresponding to the next training batch.
[0188] Each unit in the above-mentioned image feature extraction network training device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned units can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above-mentioned units.
[0189] An electronic device is also provided in an embodiment of the present application. Figure 3, which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 3 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.
[0190] like Figure 3 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 602 or programs loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing device 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0191] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.
[0192] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the image feature extraction network training methods provided in the embodiments of the present application.
[0193] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any image feature extraction network training method provided in the embodiment of the present application.
[0194] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.
[0195] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.
[0196] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0197] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).
Claims
1. A method for training an image feature extraction network, characterized in that: include: Label each image in the image dataset according to its category. Images with the same category label belong to the same object, and images with different categories belong to different objects. Constructing query samples and positive samples based on images of a first annotated category, and constructing negative samples based on images of a second annotated category, where the first annotated category is different from the second annotated category; An image feature extraction network is trained using the query sample, the positive sample, and the negative sample.
2. The method according to claim 1, characterized in that The step of constructing a query sample and a positive sample based on the image of the first labeled category includes: Selecting a first image and a second image from the images of the first labeled category; performing enhancement processing on the first image and the second image respectively to obtain a first enhanced image and a second enhanced image; The query sample and the positive sample are determined based on the first enhanced image and the second enhanced image.
3. The method according to claim 1, characterized in that The constructing of negative samples based on the image of the second labeled category includes: Selecting a third image from the images of the second labeled category; The third image is enhanced to obtain a third enhanced image, where the third image is the negative sample.
4. The method according to any one of claims 1 to 3, characterized in that The step of training an image feature extraction network using the query sample, the positive sample, and the negative sample includes: For each training batch, the query sample, the positive sample and the negative sample are used to train the image feature extraction network based on the negative sample queue using momentum contrast learning, and the negative sample queue is updated based on the negative samples in the training batch. The head of the updated negative sample queue retains the difficult samples among the negative samples in the training batch.
5. The method according to claim 4, characterized in that The updating of the negative sample queue based on the negative samples in the training batch includes: Calculating the similarity between each negative sample in the training batch and the query sample; Determine a sample category corresponding to each negative sample based on the similarity between each negative sample in the training batch and the query sample, a high similarity threshold and a low similarity threshold corresponding to the training batch, wherein the sample category includes difficult samples, common samples, and easy samples; Negative samples whose sample category is the easy sample are discarded, negative samples whose sample category is the difficult sample are inserted into the head of the negative sample queue, and negative samples whose sample category is the ordinary sample are inserted behind the difficult samples in the negative sample queue in chronological order, and the queue length of the negative sample queue is maintained at no more than the queue length threshold.
6. The method according to claim 5, characterized in that Calculating the similarity between each negative sample in the training batch and the query sample includes: Obtaining features of each negative sample in the training batch and features of the query sample; Normalizing the features of each negative sample in the training batch to obtain normalized negative sample features; Normalizing the query sample features to obtain normalized query sample features; For each negative sample in the batch, the similarity between the negative sample and the query sample is calculated based on the normalized negative sample features and the normalized query sample features.
7. The method according to claim 5, characterized in that The method of using momentum contrast learning to train the image feature extraction network based on a negative sample queue includes: Inputting the query sample into the main encoder to obtain query sample features; Inputting the positive sample into a momentum encoder to obtain positive sample features; Obtaining negative sample features in historical training batches from the negative sample queue; A preset contrastive learning loss function is used to calculate contrastive loss values between the query sample features, the positive sample features, and the negative sample features, and parameters in the main encoder and the momentum encoder are adjusted according to the contrastive loss values.
8. The method according to claim 7, characterized in that The calculating the contrast loss value between the query sample feature, the positive sample feature, and the negative sample feature using a preset contrast learning loss function includes: Based on a first contrastive learning loss function, calculating a positive sample similarity between the query sample feature and the positive sample feature, and calculating a negative sample similarity between the query sample feature and each of the negative sample features; A comparison loss value between the query sample feature, the positive sample feature representation, and the negative sample feature representation is determined according to the positive sample similarity and each of the negative sample similarities.
9. The method according to claim 7, characterized in that The calculating the contrast loss value between the query sample feature, the positive sample feature, and the negative sample feature using a preset contrast learning loss function includes: Based on the second contrastive learning loss function and the preset weights corresponding to each sample category, calculating the positive sample similarity between the query sample feature and the positive sample feature, and calculating the negative sample weighted similarity between the query sample feature and each negative sample feature; The comparison loss values among the query sample feature, the positive sample feature, and the negative sample feature are determined according to the positive sample similarity and the weighted similarities of each of the negative samples.
10. The method according to claim 7, characterized in that After calculating the contrast loss values between the query sample feature, the positive sample feature, and the negative sample feature using a preset contrast learning loss function, the method further includes: Calculating a contrast loss change rate based on the contrast loss value and the contrast loss value corresponding to the previous training batch; The high similarity threshold and the low similarity threshold corresponding to the training batch are adjusted based on the contrast loss change rate to obtain the high similarity threshold and the low similarity threshold corresponding to the next training batch.
11. An image feature extraction network training device, characterized in that: include: The category labeling unit is used to label the categories of each image in the image dataset. Images with the same category labeling belong to the same object, and images with different categories belong to different objects. a sample construction unit, configured to construct a query sample and a positive sample based on an image of a first annotated category, and to construct a negative sample based on an image of a second annotated category, wherein the first annotated category is different from the second annotated category; A training unit is used to train an image feature extraction network using the query sample, the positive sample and the negative sample.
12. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the image feature extraction network training method according to any one of claims 1 to 10.
13. An electronic device, characterized in that: comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the image feature extraction network training method as described in any one of claims 1 to 10.
14. A computer-readable storage medium, characterized in that The computer-readable storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the image feature extraction network training method as described in any one of claims 1 to 10.