A network model training method, image recognition method and related device

Abstract

The present application discloses a network model training method based on deep learning technology, including: obtaining a set of original image sample pairs and a set of newly added image sample pairs; according to the set of newly added image sample pairs, obtaining the annotation results of each original image sample pair in the set of original image sample pairs to obtain an expanded image sample pair set; constructing a triple image sample set according to the expanded image sample pair set and the set of newly added image sample pairs; obtaining the target embedded features corresponding to each sample image in each triple image sample through the network model to be trained; updating the model parameters of the network model to be trained according to the annotation results corresponding to each image sample and the target embedded features corresponding to each sample image. The present application also provides an image recognition method and related devices. The present application can realize that when training the model, there is no need to re-extract all image samples to extract embedded features, thereby saving computing resources.

Description

Technical Field

[0001] The present application relates to the field of deep learning, and in particular to a network model training method, an image recognition method, and related devices. Background Art

[0002] With the continuous development of artificial intelligence (AI) technology, image recognition based on deep learning has become the main method in the current image recognition field. Deep learning can automatically extract similar features from different images and classify them, which has promoted the development of image recognition, and unsupervised learning has become a hot topic in the field of deep learning.

[0003] At present, after the trained model is put into use, there may be sample pairs with bad case recognition errors, which are not used in the training process. Therefore, after accumulating a certain number of bad case sample pairs, these bad case sample pairs can be used to retrain the model to obtain embedding features that better support bad case sample pairs.

[0004] When updating the model, it is necessary to extract not only the embedding features of badcase sample pairs, but also the embedding features of correctly identified (goodcase) sample pairs. However, for large-scale samples, re-extracting the embedding features of all samples will consume more computing resources, resulting in low efficiency of model updating. Summary of the invention

[0005] The embodiment of the present application provides a training method of a network model, a method of image recognition, and a related device. The present application can achieve the effect of optimizing the bad case while keeping the features of the good case close. Therefore, when training the model, there is no need to re-extract all image samples to extract embedded features, thereby saving computing resources.

[0006] In view of this, the present application provides, on one hand, a method for training a network model, comprising:

[0007] Obtaining an original image sample pair set and a newly added image sample pair set, wherein the original image sample pair set includes at least two original image sample pairs used to train the original network model, and the newly added image sample pair set includes at least two newly added image sample pairs with recognition errors;

[0008] According to the newly added image sample pair set, obtaining the labeling result of each original image sample pair in the original image sample pair set, and obtaining the expanded image sample pair set, wherein the expanded image sample pair set includes the expanded image sample pairs labeled as being incorrectly recognized and the expanded image sample pairs labeled as being correctly recognized;

[0009] Constructing a triple image sample set according to the expanded image sample pair set and the newly added image sample pair set, wherein the triple image sample set includes at least one triple image sample, and the triple image sample includes at least two image samples marked as being misidentified or correctly identified;

[0010] Based on the triple image sample set, a target embedding feature corresponding to each sample image in each triple image sample is obtained through a network model to be trained, wherein the network model to be trained has the same network structure as the original network model, and the network parameters of the network model to be trained are consistent with the network parameters of the original network model;

[0011] For each triplet image sample, the model parameters of the network model to be trained are updated according to the annotation results corresponding to each image sample and the target embedded features corresponding to each sample image until the model training conditions are met and the target network model is obtained.

[0012] Another aspect of the present application provides an image recognition method, comprising:

[0013] Get the image to be queried;

[0014] Based on the image to be queried, the embedded features to be queried are obtained through the target network model, wherein the target network model is trained by adopting the above-mentioned methods;

[0015] Obtain a recall result of the image to be queried from the image library according to the embedded feature to be queried and the Q embedded features to be matched, wherein the recall result includes K images, and K and Q are both integers greater than or equal to 1;

[0016] According to the recall result, the image recognition result of the query image is determined.

[0017] On the other hand, the present application provides a model training device, comprising:

[0018] An acquisition module, used to acquire an original image sample pair set and a newly added image sample pair set, wherein the original image sample pair set includes at least two original image sample pairs used to train the original network model, and the newly added image sample pair set includes at least two newly added image sample pairs with recognition errors;

[0019] A labeling module is used to obtain the labeling result of each original image sample pair in the original image sample pair set according to the newly added image sample pair set, so as to obtain an expanded image sample pair set, wherein the expanded image sample pair set includes expanded image sample pairs labeled as being incorrectly recognized and expanded image sample pairs labeled as being correctly recognized;

[0020] A construction module, used to construct a triple image sample set according to the expanded image sample pair set and the newly added image sample pair set, wherein the triple image sample set includes at least one triple image sample, and the triple image sample includes at least two image samples marked as being misidentified or correctly identified;

[0021] The acquisition module is further used to acquire the target embedding feature corresponding to each sample image in each triple image sample through the network model to be trained based on the triple image sample set, wherein the network model to be trained has the same network structure as the original network model, and the network parameters of the network model to be trained are consistent with the network parameters of the original network model;

[0022] The training module is used to update the model parameters of the network model to be trained for each triple image sample according to the annotation results corresponding to each image sample and the target embedded features corresponding to each sample image until the model training conditions are met to obtain the target network model.

[0023] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0024] A labeling module, specifically used to obtain a first embedding feature corresponding to each original image sample through an original network model based on an original image sample pair set, wherein the original image sample pair set includes 2M original image samples, and M is an integer greater than 1;

[0025] Based on the set of newly added image sample pairs, obtaining a second embedding feature corresponding to each newly added image sample through the original network model, wherein the set of newly added image sample pairs includes 2N newly added image samples, where N is an integer greater than 1;

[0026] Performing clustering processing on the 2N newly added image samples according to the second embedded feature corresponding to each newly added image sample to obtain at least two cluster centers, wherein each cluster center corresponds to a preset pattern of recognition errors;

[0027] According to at least two cluster centers, the shortest distance corresponding to each newly added image sample in each newly added image sample pair is obtained to obtain 2N shortest distances;

[0028] According to at least two cluster centers, obtaining the distance between the first embedded feature corresponding to each original image sample and each cluster center;

[0029] According to the relationship between the distance between the first embedded feature corresponding to each original image sample and each cluster center and the distance threshold, each original image sample pair in the original image sample pair set is labeled to obtain an expanded image sample pair set, wherein the distance threshold is determined based on 2N shortest distances.

[0030] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0031] A labeling module, specifically used to take the maximum value among the 2N shortest distances as a distance threshold;

[0032] If the distance between the first embedded feature and the cluster center is less than the distance threshold, the original image sample corresponding to the first embedded feature is marked as an incorrectly recognized expanded image sample;

[0033] If the distance between the first embedded feature and the cluster center is greater than or equal to the distance threshold, the original image sample corresponding to the first embedded feature is marked as a correctly identified expanded image sample;

[0034] Two expanded image samples corresponding to the same original image sample pair are regarded as an expanded image sample pair until a set of expanded image sample pairs is obtained.

[0035] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0036] The labeling module is specifically used to label the other expanded image sample as an incorrectly recognized expanded image sample if one expanded image sample corresponding to the same original image sample pair is labeled as an incorrectly recognized expanded image sample and the other expanded image sample is labeled as an correctly recognized expanded image sample.

[0037] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0038] A construction module is specifically used to obtain similar image sample pairs from the expanded image sample pair set and the newly added image sample pair set;

[0039] Eliminate similar image sample pairs from the expanded image sample pair set and the newly added image sample pair set to obtain (M+N-1) image sample pairs, wherein the expanded image sample pair set includes M expanded image sample pairs, and the newly added image sample pair set includes N newly added image sample pairs, wherein M and N are both integers greater than 1;

[0040] Select one image sample from each of the (M+N-1) image sample pairs to obtain (M+N-1) image samples;

[0041] Calculate the distance between each image sample in the (M+N-1) image samples and the similar image sample pair to obtain (M+N-1) sample distances;

[0042] Obtain the first T sample distances with the smallest sample distance from the (M+N-1) sample distances, where T is an integer greater than or equal to 1;

[0043] The image samples corresponding to each sample distance in the T sample distances are respectively combined with similar image sample pairs to obtain T triple image samples, wherein the T triple image samples are included in the triple image sample set.

[0044] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0045] The training module is specifically used to determine the target loss function to be adopted for each triple image sample according to the labeling result corresponding to each image sample in the triple image sample;

[0046] For each triple image sample, based on the target embedding features corresponding to the sample image, the target loss function is used to determine the loss value corresponding to the triple image sample;

[0047] According to the loss value corresponding to each triplet image sample, the overall loss value is determined;

[0048] The overall loss value is adopted; the model parameters of the network model to be trained are updated.

[0049] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0050] A training module, specifically used to determine to use a first loss function if each image sample in the triplet image sample is a correctly identified image sample, wherein the first loss function includes a triplet loss function and a consistency loss function;

[0051] A training module, specifically used to obtain the original embedded features corresponding to the sample image through the original network model;

[0052] Based on the target embedding feature corresponding to the sample image, a triplet loss function included in the first loss function is used to calculate a triplet loss value;

[0053] Based on the target embedded features and the original embedded features corresponding to the sample image, a consistency loss value is calculated using a consistency loss function included in the first loss function;

[0054] The loss value corresponding to the triplet image sample is determined according to the triplet loss value and the consistency loss value.

[0055] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0056] A training module, specifically used for determining to adopt a second loss function if two image samples in the triplet image sample are correctly identified image samples and one image sample is an incorrectly identified image sample, wherein the second loss function includes an original triplet loss function, a target triplet loss function and a consistency loss function;

[0057] A training module, specifically used to obtain the original embedded features corresponding to the sample image through the original network model;

[0058] Based on the original embedded features corresponding to the sample image, the original triplet loss function included in the second loss function is used to calculate the original triplet loss value;

[0059] Based on the target embedding features corresponding to the sample image, a target triplet loss function included in the second loss function is used to calculate the target triplet loss value;

[0060] Based on the target embedded features and the original embedded features corresponding to the image sample, a consistency loss value is calculated using a consistency loss function included in the second loss function;

[0061] The loss value corresponding to the triplet image sample is determined according to the original triplet loss value, the target triplet loss value and the consistency loss value.

[0062] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0063] A training module, specifically used for determining to adopt a third loss function if two image samples in the triple image sample are image samples with recognition errors, and one image sample is a correct sample with recognition errors, wherein the third loss function includes an original triple loss function, a target triple loss function, and a consistency loss function;

[0064] A training module, specifically used to obtain the original embedded features corresponding to the sample image through the original network model;

[0065] Based on the original embedded features corresponding to the sample image, an original triplet loss function included in the third loss function is used to calculate the original triplet loss value;

[0066] Based on the target embedding features corresponding to the sample image, a target triplet loss function included in the third loss function is used to calculate the target triplet loss value;

[0067] Based on the target embedded features and the original embedded features corresponding to the image sample, a consistency loss value is calculated using a consistency loss function included in the third loss function;

[0068] The loss value corresponding to the triplet image sample is determined according to the original triplet loss value, the target triplet loss value and the consistency loss value.

[0069] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0070] A training module, specifically used for determining to adopt a fourth loss function if each image sample in the triplet image sample is an image sample with recognition error, wherein the fourth loss function includes a triplet loss function;

[0071] The training module is specifically used to calculate the loss value corresponding to the triplet image sample based on the target embedding features corresponding to the sample image, using the triplet loss function included in the fourth loss function.

[0072] Another aspect of the present application provides an image recognition device, comprising:

[0073] An acquisition module, used to acquire the image to be queried;

[0074] The acquisition module is further used to acquire the embedded features to be queried through the target network model based on the image to be queried, wherein the target network model is trained by using the training method of any one of claims 1 to 10 above;

[0075] The acquisition module is further used to acquire the recall result of the image to be queried from the image library according to the embedded feature to be queried and the Q embedded features to be matched, wherein the recall result includes K images, and K and Q are both integers greater than or equal to 1;

[0076] The determination module is used to determine the image recognition result of the image to be queried according to the recall result.

[0077] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0078] An acquisition module is specifically used to acquire a first feature distance between the embedded feature to be queried and each of the embedded features to be matched in the Q embedded features to be matched, and obtain Q first feature distances, wherein each embedded feature to be matched is a cluster center of a cluster;

[0079] Obtain at least one clustering result from the Q clustering results according to the Q first feature distances;

[0080] Obtaining a second feature distance between the embedded feature to be queried and each image in at least one clustering result;

[0081] According to the second feature distance between each image, K images in the image library are used as the recall results of the query image.

[0082] Another aspect of the present application provides a computer device, including: a memory, a processor, and a bus system;

[0083] Wherein, the memory is used to store programs;

[0084] The processor is used to execute the program in the memory, and the processor is used to execute the above-mentioned methods according to the instructions in the program code;

[0085] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0086] Another aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.

[0087] Another aspect of the present application provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided by the above aspects.

[0088] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0089] In an embodiment of the present application, a training method for a network model is provided. First, an original image sample pair set and a newly added image sample pair set are obtained. Then, the annotation result of each original image sample pair in the original image sample pair set can be obtained according to the newly added image sample pair set to obtain an expanded image sample pair set. Based on this, a triple image sample set is constructed according to the expanded image sample pair set and the newly added image sample pair set, where the triple image sample includes at least two image samples annotated as misidentified or correctly identified image samples. Then, based on the triple image sample set, the target embedding feature corresponding to each sample image in each triple image sample is obtained through the network model to be trained. Thus, according to the annotation result corresponding to each image sample and the target embedding feature corresponding to each sample image, the model parameters of the network model to be trained are updated until the model training conditions are met, and the target network model can be trained to obtain. In the above manner, the original image sample pair set is annotated using the newly added image sample pair set, so as to obtain the image sample pairs that are correctly identified (i.e., goodcase) and the image sample pairs that are incorrectly identified (i.e., badcase). Since goodcase can make the feature distribution consistent and badcase can perform effective metric learning, it is possible to achieve the effect of optimizing badcase while keeping the features of goodcase close. Therefore, using these image sample pairs (i.e., goodcase and badcase) to update the model does not require re-extracting all image samples to extract embedded features, thereby saving computing resources and improving the efficiency of model updating. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 This is a schematic diagram of the architecture of the network model training system in the embodiment of the present application;

[0091] Figure 2 A schematic diagram of the architecture of an image recognition system in an embodiment of the present application;

[0092] Figure 3 A flowchart of a network model training method in an embodiment of the present application;

[0093] Figure 4 A schematic diagram of the structure of the original network model in the embodiment of the present application;

[0094] Figure 5 A schematic diagram of generating an expanded sample pair based on an original sample pair and a newly added sample pair in an embodiment of the present application;

[0095] Figure 6 A schematic diagram of the structure of the original network model and the network model to be trained in the embodiment of the present application;

[0096] Figure 7 A schematic diagram of the structure of a residual module in an embodiment of the present application;

[0097] Figure 8 A schematic diagram of a flow chart of an image recognition method in an embodiment of the present application;

[0098] Fig. 9 This is a schematic diagram of bucket-based retrieval in an embodiment of the present application;

[0099] Fig.10 A schematic diagram of implementing the training and application of a network model in an embodiment of the present application;

[0100] Fig.11 A schematic diagram of a model training device in an embodiment of the present application;

[0101] Fig.12 A schematic diagram of an image recognition device in an embodiment of the present application;

[0102] Fig.13 A schematic diagram of the structure of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0103] The embodiment of the present application provides a training method of a network model, a method of image recognition, and a related device. The present application can achieve the effect of optimizing the bad case while keeping the features of the good case close. Therefore, when training the model, there is no need to re-extract all image samples to extract embedded features, thereby saving computing resources.

[0104] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0105] The embedding features of images generated based on deep learning can be used in fields such as image retrieval and image recognition. For example, search for similar images, search for identical images, search for product images, and search for picture book images. For example, a user can take a photo of a picture book, and automatically retrieve the corresponding cover or page in the database, further realizing functions such as picture book follow-up reading. For example, the pictures taken by the user are searched in the product database to find the same or similar products, sell the products or recommend related products, and improve the convenience of product search and search. For example, search the image database to see if there are identical or highly similar images, implement image deduplication or filtering within the system, and avoid waste of resources caused by duplicate content.

[0106] Normally, a network model is needed to extract the embedding features of an image. Therefore, the present application provides a training method for a network model, which can screen out goodcases based on newly added image sample pairs and original image sample pairs, keep their feature distribution unchanged, screen out badcases for effective metric learning, and achieve goodcases before and after fine-tuning to keep features close, while badcases are optimized. Specifically, the present application proposes a learning method for distinguishing badcases from goodcases, and discovers badcases after the original network model is obtained by training the original image sample pair set. This type of badcase occurs due to the small number of badcases in the original image sample pair set and the lack of robust embedding feature learning. Based on this, after collecting a set of newly added image sample pairs for badcases, the set of newly added image sample pairs can be used to annotate the original image sample pair set. While not changing the representation of the goodcase samples, the optimization of the badcase representation is achieved, so that the goodcase performance is consistent with the original network model, and the badcase effect is improved. Then, in the application, we only need to find the bad cases and update these bad case representations under the target network model, so as to avoid time-consuming large-scale inventory updates and achieve better bad case optimization effects under limited updates.

[0107] Based on this, the following will introduce the network model training system and image recognition system respectively. Figure 1 , Figure 1This is an architectural diagram of a network model training system in an embodiment of the present application. As shown in the figure, the network model training system includes a server and a terminal device, and the client is deployed on the terminal device, wherein the client can be run on the terminal device in the form of a browser, or in the form of an independent application (application, APP), etc. The specific presentation form of the client is not limited here. The labeling personnel can label the image sample pairs on the client provided by the terminal device, for example, label a certain image sample pair as "goodcase" or "badcase". The labeled image sample pairs are uploaded to the server, and the server performs model training based on the labeled image sample pairs to obtain the target network model.

[0108] See also Figure 2 , Figure 2 This is a schematic diagram of the architecture of the image recognition system in the embodiment of the present application. As shown in the figure, the image recognition system includes a server and a terminal device, and the client is deployed on the terminal device. The user can upload the image to be queried to the server on the client provided by the terminal device. The server calls the trained target network model to extract the embedding features of the image to be queried, and then matches it with the embedding features of other images in the database. Finally, the server returns the image recognition result to the terminal device.

[0109] The server involved in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal device can be a smart phone, tablet computer, laptop computer, PDA, personal computer, smart TV, smart watch, vehicle-mounted device, wearable device, etc., but is not limited to this. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this. The number of servers and terminal devices is also not limited. The solution provided in this application can be completed independently by the terminal device, or by the server, or by the terminal device and the server in cooperation, and this application does not make specific limitations on this.

[0110] Based on this, it can be seen that the network model training method provided in this application involves machine learning technology. Among them, machine learning is a multi-field interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.

[0111] Combined with the above introduction, the training method of the network model in this application will be introduced below. Please refer to Figure 3 , an embodiment of the network model training method in the embodiment of the present application includes:

[0112] 110. Obtain an original image sample pair set and a newly added image sample pair set, wherein the original image sample pair set includes at least two original image sample pairs used to train the original network model, and the newly added image sample pair set includes at least two newly added image sample pairs with recognition errors;

[0113] In one or more embodiments, the network model training device obtains a set of original image sample pairs and a set of newly added image sample pairs. Wherein, the set of original image sample pairs includes original image sample pairs marked as "similar", and each original image sample pair is composed of two original image samples. And the set of newly added image sample pairs includes newly added image sample pairs marked as "similar" and marked as "recognition error (i.e., badcase)". Since each newly added image sample pair is composed of two newly added image samples, each newly added image sample is marked as "badcase". Wherein, for online badcase styles (e.g., styles of basketball game types), a batch of image positive sample pairs similar to the badcase style are collected as newly added image sample pairs. The collected newly added image sample pairs can be considered as badcase samples in the original image sample pair set, and the features of such badcase samples need to be updated to better representations. Assuming that there are 100 known badcase styles, 100 sample pairs are collected for each style, that is, the set of newly added image sample pairs includes 10,000 newly added image sample pairs.

[0114] Specifically, for ease of explanation, see Figure 4 , Figure 4This is a structural diagram of the original network model in the embodiment of the present application. As shown in the figure, the original image sample pair is input into the original network model, and the original network model outputs the embedding features of each original image sample. The embedding features of two original image samples can characterize the similarity between the images. The original network model can be trained using triples, wherein the triplet includes similar original image sample pairs and an original image sample with a high degree of similarity to the original image sample pair. Based on this, the original network model can be trained using triplet loss.

[0115] It should be noted that the network model training device can be deployed on a server or a terminal device, or can be deployed in a system composed of a server and a terminal device. The original image sample pair set and the newly added image sample pair set can be extracted from a large general object recognition open source dataset (ImageNet), or can be collected through other channels. The original network model can use convolutional neural networks (CNN) and fully connected layers (FC), or other types of network structures.

[0116] 120. According to the newly added image sample pair set, obtain the labeling result of each original image sample pair in the original image sample pair set to obtain an expanded image sample pair set, wherein the expanded image sample pair set includes expanded image sample pairs labeled as being incorrectly recognized and expanded image sample pairs labeled as being correctly recognized;

[0117] In one or more embodiments, since the newly added image sample pairs are all "badcase", it is possible not only to find original image sample pairs similar to "badcase" from the original image sample pair set, and to mark these original image sample pairs (for example, marking them as "1" to represent "badcase"). It is also possible to find original image sample pairs that are not similar to "badcase" from the original image sample pair set, and to mark these original image sample pairs (for example, marking them as "0" to represent "goodcase"). The marked original image sample pairs are the expanded image sample pairs.

[0118] For illustration purposes, see Figure 5 , Figure 5This is a schematic diagram of generating an expanded sample pair based on the original sample pair and the newly added sample pair in an embodiment of the present application. As shown in the figure, the original image sample pair is only labeled with the "similar" label, while the newly added image sample pair is not only labeled with the "similar" label, but also labeled with "recognition error (i.e., badcase)". Based on this, using these newly added image sample pairs, the original image sample pairs can be labeled twice, that is, labeled as "recognition error (i.e., badcase)" or "recognition correct (i.e., goodcase)", thereby obtaining the expanded image sample pairs.

[0119] 130. Construct a triple image sample set according to the expanded image sample pair set and the newly added image sample pair set, wherein the triple image sample set includes at least one triple image sample, and the triple image sample includes at least two image samples marked as being misidentified or correctly identified;

[0120] In one or more embodiments, the expanded image sample pair set and the newly added image sample pair set are used as a common training set, and then at least one triple image sample is selected from each of the image sample pairs, thereby obtaining a triple image sample set. Each triple image sample consists of three image samples, wherein at least two of the three image samples are labeled as "badcase" image samples or "goodcase" image samples.

[0121] 140. Based on the triple image sample set, obtaining the target embedding feature corresponding to each sample image in each triple image sample through the network model to be trained, wherein the network model to be trained has the same network structure as the original network model, and the network parameters of the network model to be trained are consistent with the network parameters of the original network model;

[0122] In one or more embodiments, each triple image sample in the triple image sample set is input into the network model to be trained, and the network model to be trained outputs the target embedding features corresponding to each sample image in each triple image sample. The network model to be trained and the original network model have the same network structure and the same network parameters, so both the network model to be trained and the original network model can be understood as "old models".

[0123] It can be seen that the present application retrains the network model by adding triplet image samples labeled as “badcase” and / or “goodcase”, thereby making the network model more robust and solving the data fine-tuning problem.

[0124] 150. For each triplet image sample, the model parameters of the network model to be trained are updated according to the annotation results corresponding to each image sample and the target embedded features corresponding to each sample image, until the model training conditions are met to obtain the target network model.

[0125] In one or more embodiments, for ease of description, see Figure 6 , Figure 6 This is a structural diagram of the original network model and the network model to be trained in the embodiment of the present application. As shown in the figure, the target embedding features corresponding to each sample image in the triple image sample are obtained through the "old model". During training, the labeling results (i.e., "badcase" or "goodcase") and the target embedding features corresponding to each image sample in the triple image sample are calculated for loss, and are transmitted back to the network model to be trained to update the model parameters. When the model training conditions are met, the target network model is obtained. The target network model can be understood as a "new model".

[0126] It can be seen that the above model update process belongs to fine-tuning learning, that is, for an already trained model, when only part of the network parameters need to be updated, or new data is added to retrain the network parameters, it is all fine-tuning learning. The purpose of fine-tuning is to obtain a better model based on the original model parameters, or to obtain certain downstream tasks (for example, adding a downstream classification task to the model).

[0127] In the embodiment of the present application, a training method of a network model is provided, and in the above-mentioned manner, the original image sample pair set is annotated using the newly added image sample pair set, so as to obtain the image sample pair (i.e., goodcase) with correct recognition and the image sample pair (i.e., badcase) with wrong recognition. Since goodcase can make the feature distribution consistent, and badcase can perform effective metric learning, it is possible to achieve the effect of making badcase optimized while keeping the feature close to goodcase. Therefore, these image sample pairs (i.e., goodcase and badcase) are used to update the model, and there is no need to re-extract all image samples to extract embedded features, thereby saving computing resources and improving the efficiency of model updating.

[0128] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in an optional embodiment provided by the embodiment of the present application, according to the newly added image sample pair set, the annotation result of each original image sample pair in the original image sample pair set is obtained to obtain the expanded image sample pair set, which may specifically include:

[0129] Based on the original image sample pair set, obtaining a first embedding feature corresponding to each original image sample through the original network model, wherein the original image sample pair set includes 2M original image samples, and M is an integer greater than 1;

[0130] Based on the set of newly added image sample pairs, obtaining a second embedding feature corresponding to each newly added image sample through the original network model, wherein the set of newly added image sample pairs includes 2N newly added image samples, where N is an integer greater than 1;

[0131] Performing clustering processing on the 2N newly added image samples according to the second embedded feature corresponding to each newly added image sample to obtain at least two cluster centers, wherein each cluster center corresponds to a preset pattern of recognition errors;

[0132] According to at least two cluster centers, the shortest distance corresponding to each newly added image sample in each newly added image sample pair is obtained to obtain 2N shortest distances;

[0133] According to at least two cluster centers, obtaining the distance between the first embedded feature corresponding to each original image sample and each cluster center;

[0134] According to the relationship between the distance between the first embedded feature corresponding to each original image sample and each cluster center and the distance threshold, each original image sample pair in the original image sample pair set is labeled to obtain an expanded image sample pair set, wherein the distance threshold is determined based on 2N shortest distances.

[0135] In one or more embodiments, a method of annotating original image sample pairs is introduced. It can be seen from the above embodiments that each image sample pair is composed of two image samples, that is, the original image sample pair set includes 2M original image samples, and the newly added image sample pair set includes 2N newly added image samples, and M and N are both integers greater than 1. The original image sample pair is annotated with a "similar" label, so the original image sample pair belongs to a positive sample pair. Considering the problem of incomplete data coverage of the original image sample pair, the image retrieval effect of some image styles is poor. For example, the image style retrieval effect of "basketball game type" is poor. This is because the features of the basketball court are not distinguishable, and the image area occupied by a single person is small. The uniforms worn by the characters are relatively similar, and the audience cannot provide distinguishable features, so it is difficult to distinguish. The original image sample pair may also contain a small number of badcase preset styles (for example, forest style or black and white comic style, etc.), but due to the small amount of data, poor learning occurs. Therefore, different losses will be used to learn such samples during subsequent model fine-tuning.

[0136] Specifically, two original image samples belonging to the same original image sample pair are respectively input into the original network model, and the original network model outputs the first embedding feature of each original image sample. Similarly, two newly added image samples belonging to the same newly added image sample pair are respectively input into the original network model, and the original network model outputs the second embedding feature of each newly added image sample. Based on this, according to the second embedding feature of the newly added image sample, a clustering algorithm is used to cluster the preset styles of each badcase, and at least two cluster centers are obtained.

[0137] Exemplarily, the clustering algorithms involved in the present application include but are not limited to K-means clustering algorithm (k-means clustering algorithm, kmeans), spectral clustering, and aggregation based on embedding density (for example, counting the number of samples existing in the vicinity of a sample in the embedding space, when the number of samples is greater than the average number of samples per cluster, the sample is used as a cluster center), etc.

[0138] Based on this, the distance between the two newly added image samples and the nearest cluster center of each newly added image sample pair is calculated respectively. Since there are 2N newly added image samples in total, 2N shortest distances can be obtained. A distance threshold can be determined based on the 2N shortest distances. In addition, the distance between each first embedding feature and each cluster center needs to be calculated respectively. Therefore, based on the size relationship between these distances and the distance threshold, each original image sample pair can be labeled, and finally an expanded image sample pair set is obtained.

[0139] Secondly, in an embodiment of the present application, a method for annotating original image sample pairs is provided. Through the above method, a newly added image sample pair set can be used to automatically annotate each original image sample pair without manual annotation, thereby saving manual annotation costs and improving annotation efficiency.

[0140] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in an optional embodiment provided by the embodiment of the present application, each original image sample pair in the original image sample pair set is labeled according to the relationship between the distance between the first embedded feature corresponding to each original image sample and each cluster center and the distance threshold, to obtain an expanded image sample pair set, which may specifically include:

[0141] The maximum value among the 2N shortest distances is used as the distance threshold;

[0142] If the distance between the first embedded feature and the cluster center is less than the distance threshold, the original image sample corresponding to the first embedded feature is marked as an incorrectly recognized expanded image sample;

[0143] If the distance between the first embedded feature and the cluster center is greater than or equal to the distance threshold, the original image sample corresponding to the first embedded feature is marked as a correctly identified expanded image sample;

[0144] Two expanded image samples corresponding to the same original image sample pair are regarded as an expanded image sample pair until a set of expanded image sample pairs is obtained.

[0145] In one or more embodiments, a method for annotating an expanded image sample pair is introduced. As can be seen from the above embodiments, the distance threshold can be determined based on 2N shortest distances. In this application, the maximum value of the 2N shortest distances is used as the distance threshold. Alternatively, the average value of the 2N shortest distances can also be used as the distance threshold.

[0146] Specifically, assuming there are 1000 cluster centers, the distance between each first embedded feature and the 1000 cluster centers is calculated, and the original image samples whose distance is less than the distance threshold (d_badcase_thr) are marked as badcase, and the original image samples whose distance is greater than or equal to the distance threshold (d_badcase_thr) are marked as goodcase.

[0147] Again, in an embodiment of the present application, a method for annotating and obtaining an expanded image sample pair is provided. Through the above method, a distance threshold can be used to determine whether the original image sample belongs to a badcase. The maximum value of the shortest distance is used as the distance threshold. The distance threshold can be maximized under reasonable circumstances, thereby screening out more expanded image sample pairs marked as badcases.

[0148] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in an optional embodiment provided by the embodiment of the present application, two expanded image samples corresponding to the same original image sample pair are taken as an expanded image sample pair, which may specifically include:

[0149] If one augmented image sample corresponding to the same original image sample pair is labeled as an incorrectly recognized augmented image sample, and the other augmented image sample is labeled as a correctly recognized augmented image sample, the other augmented image sample is labeled as an incorrectly recognized augmented image sample.

[0150] In one or more embodiments, a method for correcting the annotation results is introduced. Since the original image sample pair belongs to two original image samples labeled as "similar", the same original image sample pair should show the same characteristics, for example, both are "badcase" or "goodcase". However, there is a deviation between the two image samples in the image sample pair, or insufficient model learning, etc., which may cause one expanded image sample corresponding to the same original image sample pair to be labeled as "goodcase" and the other expanded image sample to be labeled as "badcase". Based on this, it is necessary to label both expanded image samples as "badcase".

[0151] Furthermore, in an embodiment of the present application, a method for correcting the annotation results is provided. Through the above method, the annotation results can be automatically corrected, which not only improves the accuracy of the annotation results, but also increases the number of bad cases to a certain extent, thereby achieving the purpose of maximum training and avoiding sample pair mismatch during model training.

[0152] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in an optional embodiment provided by the embodiment of the present application, a triple image sample set is constructed according to the expanded image sample pair set and the newly added image sample pair set, which may specifically include:

[0153] Obtain similar image sample pairs from the expanded image sample pair set and the newly added image sample pair set;

[0154] Eliminate similar image sample pairs from the expanded image sample pair set and the newly added image sample pair set to obtain (M+N-1) image sample pairs, wherein the expanded image sample pair set includes M expanded image sample pairs, and the newly added image sample pair set includes N newly added image sample pairs, wherein M and N are both integers greater than 1;

[0155] Select one image sample from each of the (M+N-1) image sample pairs to obtain (M+N-1) image samples;

[0156] Calculate the distance between each image sample in the (M+N-1) image samples and the similar image sample pair to obtain (M+N-1) sample distances;

[0157] Obtain the first T sample distances with the smallest sample distance from the (M+N-1) sample distances, where T is an integer greater than or equal to 1;

[0158] The image samples corresponding to each sample distance in the T sample distances are respectively combined with similar image sample pairs to obtain T triple image samples, wherein the T triple image samples are included in the triple image sample set.

[0159] In one or more embodiments, a method for constructing a triple image sample is introduced. After merging the expanded image sample pair set and the newly added image sample pair set, a badcase image sample pair marked as "1" and a goodcase image sample pair marked as "0" can be obtained. In addition, all badcase image samples can also be recorded in the badcase sample database for subsequent training. In model fine-tuning, triple image samples can be mined for each batch of training sets. For ease of explanation, the following will be introduced by generating a triple image sample.

[0160] Specifically, a similar image sample pair (i.e., two image samples marked as "similar") is extracted from a batch training set (i.e., M expanded image sample pairs and N newly added image sample pairs). Therefore, the similar image sample pair may be an expanded image sample pair or a newly added image sample pair, which is not limited here. Assuming that a batch has (M+N) image sample pairs, for similar image sample pairs, an image sample is randomly selected from each image sample pair of the remaining (M+N-1) image sample pairs, that is, (M+N-1) image samples are obtained. Based on this, the distance between each image sample in the (M+N-1) image samples and the similar image sample pair is calculated respectively. For example, the similar image sample pair includes image sample A and image sample B, and the (M+N-1) image samples include image sample C. Therefore, the sample distance corresponding to image sample C may include the sum of the distances between image sample C and image sample A and image sample B, or the average value of the distances. Based on this, (M+N-1) sample distances are obtained.

[0161] Arrange the (M+N-1) sample distances from small to large and take the image sample corresponding to the first T sample distances (for example, image sample C) as a negative sample, and form a triple image sample with a similar image sample pair. Assuming T is 10, each image sample generates 10 triple image samples, and the entire batch can obtain 10*(M+N) triple image samples.

[0162] Since the image samples corresponding to the first T sample distances may come from newly added image sample pairs or expanded image sample pairs, and these image sample pairs include "badcase" and "goodcase", the image samples corresponding to the first T sample distances may be "badcase" or "goodcase". That is, the triple image samples include four types of composition, namely:

[0163] In case 1, the triplet image samples include “goodcase”, “goodcase” and “goodcase”;

[0164] In case 2, the triplet image samples include “goodcase”, “goodcase” and “badcase”;

[0165] Case 3, the triplet image samples include “badcase”, “badcase” and “goodcase”;

[0166] In case 4, the triplet image samples include “badcase”, “badcase”, and “badcase”.

[0167] It should be noted that the triplet image samples used in training the original network model can also be generated in a similar manner, which will not be elaborated here.

[0168] Secondly, in the embodiment of the present application, a method for constructing triple image samples is provided, through which the expanded image sample pair set and the newly added image sample pair set are merged into a training set, thereby achieving the purpose of expanding the badcase data set. In addition, constructing triple image samples is used for subsequent model training to improve the reliability of model training.

[0169] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in an optional embodiment provided by the embodiment of the present application, for each triple image sample, according to the annotation result corresponding to each image sample and the target embedding feature corresponding to each sample image, the model parameters of the network model to be trained are updated, which may specifically include:

[0170] For each triplet image sample, determine the target loss function to be used according to the labeling result corresponding to each image sample in the triplet image sample;

[0171] For each triple image sample, based on the target embedding features corresponding to the sample image, the target loss function is used to determine the loss value corresponding to the triple image sample;

[0172] According to the loss value corresponding to each triplet image sample, the overall loss value is determined;

[0173] The overall loss value is adopted; the model parameters of the network model to be trained are updated.

[0174] In one or more embodiments, a method for updating a network model to be trained based on triplet image samples is introduced. As can be seen from the aforementioned embodiments, the network model to be trained has the same network structure and network parameters as the original network model, so the model parameters of the original network model can be used as the initialization model parameters of the network model to be trained.

[0175] Specifically, the original network model can use the residual network-101 (ResidualNetwork-101, ResNet-101) trained based on ImageNet. In practical applications, networks such as ResNet-34 or ResNet-50 can also be used. For ease of explanation, please refer to Table 1, which shows the feature module structure of the ResNet-101 model.

[0176] Table 1

[0177]

[0178] As shown in Table 1, the convolutional layers (Conv1) to (Conv5) use the parameters of ResNet-101 pre-trained on the ImageNet dataset. Taking convolutional layer 2 as an example, it includes 3 residual blocks. Figure 7 , Figure 7 This is a structural diagram of the residual module in the embodiment of the present application. As shown in the figure, the feature map of 256-dimensional channels is reduced to a feature map of 64-dimensional channels through the first 1×1 convolution layer in the residual module. Then, the second 3×3 convolution layer in the residual module is used to convolve the feature map of 64-dimensional channels. Finally, the third 1×1 convolution layer in the residual module is used to restore the feature map of 64-dimensional channels to a feature map of 256-dimensional channels.

[0179] After the convolutional layer (Conv1) to the convolutional layer (Conv5), the pooling layer, the fully connected layer and the normalization layer are added. The model parameters of these network layers need to be learned. Please refer to Table 2, which is the embedding learning module based on ResNet-101.

[0180] Table 2

[0181]

[0182] Among them, the embedding layer is used to generate image similarity representation. The basic features and the embedding layer can use a learning rate of lr1=0.0005. During the training process, the full amount of data is iterated in rounds (epochs), and each epoch iteration processes a full amount of samples (i.e., a full amount of expanded image sample pairs and newly added image sample pairs). In each epoch iteration, each batch size (batchsize) image samples in the full amount of image samples are taken as a batch and divided into Nb batches. In each batch, a forward calculation is performed on an input image sample to obtain a prediction result, that is, the output target embedding feature. In a batch, based on the labeling results corresponding to different image samples in the triple image sample (for example, "badcase" or "goodcase"), the target loss function used is determined, and then the target loss function is used to determine the loss value corresponding to the triple image sample.

[0183] After calculating the loss of all triple image samples, the overall loss value is obtained. Stochastic gradient descent (SGD) is used to perform gradient backward calculation on the overall loss value and obtain the updated values ​​of all model parameters, and the model parameters of the network model to be trained are updated. When the iteration threshold (for example, 10 times) is reached, or the overall loss value converges (for example, it has not decreased for 10 consecutive rounds), it means that the model training conditions are met, and the model training is stopped at this time.

[0184] It should be noted that the training method type of the original network model first requires parameter initialization. The model parameters of ResNet-101 pre-trained on the ImageNet dataset can be used. The newly added embedding layer can be initialized with a Gaussian distribution with a variance of 0.01 and a mean of 0. At the same time, the learning parameters and learning rate need to be set. During the training process, the full amount of data is iterated epoch, and the full amount of samples is processed once in each epoch iteration. In each epoch iteration, each batchsize image sample in the full amount of original image sample pairs is taken as a batch and divided into Np batches. In each batch, a forward calculation is performed on an input original image sample to obtain the prediction result, that is, the output embedding feature. In a batch, the triple loss function used determines the loss value corresponding to the triple image sample. After the loss calculation is performed on all triple image samples, the overall loss value is obtained. The overall loss value is gradient-backward calculated using SGD to obtain the updated value of all model parameters, and the model parameters of the network model to be trained are updated. When the model training conditions are met, the model training is stopped.

[0185] Secondly, in the embodiment of the present application, a method for updating the network model to be trained based on triple image samples is provided. Through the above method, in the model training stage, triple learning is performed using the bad cases that are reasonably mined, and the features of the good cases in the database are not updated. During the learning, the distribution of the new features of the good cases (i.e., the embedding of the expanded image samples) is made the same as the distribution of the old features (i.e., the embedding of the original image samples), so that better measurement effects can be obtained in the application without updating the good cases. For the bad cases in the database, their features are updated, and the measurement effects between them and the old features of the good cases are satisfied at the same time.

[0186] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in an optional embodiment provided by the embodiment of the present application, according to the labeling result corresponding to each image sample in the triple image sample, determining to adopt the target loss function may specifically include:

[0187] If each image sample in the triplet image sample is an image sample that is correctly identified, it is determined to adopt a first loss function, wherein the first loss function includes a triplet loss function and a consistency loss function;

[0188] Based on the target embedding features corresponding to the sample image, the target loss function is used to determine the loss value corresponding to the triple image sample, which may specifically include:

[0189] Obtain the original embedded features corresponding to the sample image through the original network model;

[0190] Based on the target embedding feature corresponding to the sample image, a triplet loss function included in the first loss function is used to calculate a triplet loss value;

[0191] Based on the target embedded features and the original embedded features corresponding to the sample image, a consistency loss value is calculated using a consistency loss function included in the first loss function;

[0192] The loss value corresponding to the triplet image sample is determined according to the triplet loss value and the consistency loss value.

[0193] In one or more embodiments, a method for calculating a loss value is introduced. It can be seen from the above embodiments that in one case, the triplet image sample includes goodcase, goodcase and goodcase.

[0194] Specifically, if each image sample in the triplet image sample is a good case, the consistency loss function loss of metric learning can be used con, loss con It can make the new and old features consistent. At the same time, for the good case of the new input, it can satisfy the triplet loss function (triplet-loss) loss under the new features. tri The requirement is that α = 0.5. The margin of the triplet loss function is the same as that of the original network model, which is 0.8.

[0195] Based on this, the first loss function can be expressed as:

[0196] L1=loss tri-new0.8 +α*loss con:apn ;

[0197] Among them, L1 represents the loss value of the triple image sample. tri-new0.8 represents the triple loss function with a margin of 0.8. con:apn represents the consistency loss function. α represents the weight value, for example, α = 0.5.

[0198] More specifically, loss con As a loss function for the consistency of feature distribution, this application takes the KL divergence loss (Kullback-Leibler, KL-loss) function as an example to measure the similarity between two probability distributions. The closer the probability distribution is to the KL divergence, the smaller it is.

[0199] The KL-loss function can be expressed as:

[0200]

[0201] Among them, p(x i ) represents the original network model for the sample image x i The original embedding features predicted. q(x i ) represents the network model to be trained for the sample image x i The predicted target embedding features, the original embedding features can be expressed as a 128-dimensional vector, and the target embedding features can be expressed as a 128-dimensional vector.

[0202] Consistency loss function loss con:apn Indicates that all three image samples in the triple image sample need to calculate KL-loss. Assume that a triple image sample includes image sample x a , image sample x b and image sample x c , then based on the image sample x aThe original embedding features and target embedding features of the image sample x are calculated. a KL-loss of . Similarly, calculate the image samples x b KL-loss and image sample x c Therefore, according to the KL-loss of the three image samples in the triple image sample, the consistency loss value loss can be obtained con:apn .

[0203] The triplet loss function can be expressed as:

[0204] loss tri-new0.8 =max(||x anew -x pnew ||-||x anew -x nnew ||+β,0);

[0205] Among them, loss tri-new0.8 Represents the triple loss value. x anew Represents the image sample x in the triplet image sample a The corresponding target embedding feature. pnew Represents the image sample x that is a positive sample in the triplet image sample p The corresponding target embedding feature. nnew Represents the image sample x that is a negative sample in the triplet image sample n The corresponding target embedding feature. It can be understood that the positive sample is the same as x a The negative sample is an independent image sample added to the triplet image sample later. β represents margin, which can be set to 0.8. anew -x pnew || represents the L2 distance between two target embedding features, ||x anew -x nnew || represents the L2 distance between two target embedding features. The purpose of the triplet loss function is to make the image sample x a With image sample x n The distance is greater than the distance image sample x p The distance is greater than 0.4.

[0206] It should be noted that the weight value α can also be adjusted to 0.1, that is, the proportion of the triplet loss function is increased, which is not limited here.

[0207] Again, in an embodiment of the present application, a method for calculating the loss value is provided. Through the above method, the loss value is calculated for the three "goodcase" triplet image samples in a targeted manner, thereby improving the reliability and flexibility of the calculation.

[0208] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in an optional embodiment provided by the embodiment of the present application, according to the labeling result corresponding to each image sample in the triple image sample, determining to adopt the target loss function may specifically include:

[0209] If two image samples in the triplet image sample are correctly identified image samples, and one image sample is an incorrectly identified image sample, it is determined to adopt a second loss function, wherein the second loss function includes an original triplet loss function, a target triplet loss function, and a consistency loss function;

[0210] Based on the target embedding features corresponding to the sample image, the target loss function is used to determine the loss value corresponding to the triple image sample, which may specifically include:

[0211] Obtain the original embedded features corresponding to the sample image through the original network model;

[0212] Based on the original embedded features corresponding to the sample image, the original triplet loss function included in the second loss function is used to calculate the original triplet loss value;

[0213] Based on the target embedding features corresponding to the sample image, a target triplet loss function included in the second loss function is used to calculate the target triplet loss value;

[0214] Based on the target embedded features and the original embedded features corresponding to the image sample, a consistency loss value is calculated using a consistency loss function included in the second loss function;

[0215] The loss value corresponding to the triplet image sample is determined according to the original triplet loss value, the target triplet loss value and the consistency loss value.

[0216] In one or more embodiments, a method for calculating a loss value is introduced. It can be seen from the above embodiments that in one case, the triplet image sample includes goodcase, goodcase and badcase.

[0217] Specifically, for goodcase, we hope that the embeddings predicted by the original network model and the network model to be trained are as consistent as possible. At the same time, for the triplet loss function, it is necessary not only to satisfy the triplet loss function under the target embedding feature, but also to satisfy the triplet loss function under the image sample x. aThe original embedding features, image samples x p The original embedding features and image samples x n The triple loss constraint under the target embedding feature. Therefore, two triple loss functions are needed here, namely the target triple loss function loss tri-new And the original triple loss function loss tri-old Among them, the target triple loss function loss tri-new Indicates that the three image samples all use the target embedding feature to calculate the metric learning margin of 0.8. The original triplet loss function loss tri-old Represents image sample x a and image sample x p The original embedding feature is used to calculate the metric learning margin of 0.6, and the image sample x n The target embedding features are used for calculation. This hybrid metric calculation can distinguish the good cases without updating features from the negative samples consisting of bad cases under the target embedding features.

[0218] Based on this, the second loss function can be expressed as:

[0219] L2=loss tri-old0.6 +loss tri-new0.8 +α*loss con:ap ;

[0220] Among them, L2 represents the loss value of the triple image sample. tri-old0.6 Represents the original triplet loss function with a margin of 0.6. tri-new0.8 represents the target triplet loss function with a margin of 0.8. con:ap represents the consistency loss function. α represents the weight value, for example, α = 0.5.

[0221] More specifically, as described in the above embodiment, the consistency loss function loss con:ap Indicates that both image samples in the triple image sample need to calculate KL-loss. Assume that a triple image sample includes image sample x a and image sample x b , then based on the image sample x a The original embedding features and target embedding features of the image sample x are calculated. a KL-loss of . Similarly, calculate the image samples x bTherefore, according to the KL-loss of two image samples in the triple image sample, the consistency loss value loss can be obtained con:ap .

[0222] The target triplet loss function can be expressed as:

[0223] loss tri-new0.8 =max(||x anew -x pnew ||-||x anew -x nnew ||+β,0);

[0224] Among them, loss tri-new0.8 Represents the target triplet loss value. x anew Represents the image sample x in the triplet image sample a The corresponding target embedding feature. pnew Represents the image sample x that is a positive sample in the triplet image sample p The corresponding target embedding feature. nnew Represents the image sample x that is a negative sample in the triplet image sample n The corresponding target embedding feature. β represents margin, which can be set to 0.8.

[0225] The original triplet loss function can be expressed as:

[0226] loss tri-old0.6 =max(||x aold -x pold ||-||x aold -x nnew ||+β,0);

[0227] Among them, loss tri-new0.8 Represents the target triplet loss value. x aold Represents the image sample x in the triplet image sample a The corresponding original embedding feature. pold Represents the image sample x that is a positive sample in the triplet image sample p The corresponding original embedding feature. nnew Represents the image sample x that is a negative sample in the triplet image sample n The corresponding target embedding feature. β represents margin, which can be set to 0.6.

[0228] It should be noted that the weight value α can also be adjusted to 0.1, that is, to increase the proportion of the triplet loss function, which is not limited here. The margin of the original network model can be set not only to 0.6, but also to other values, such as 1. The margin of the network model to be trained can be set not only to 0.8, but also to other values, such as 1.6. Usually, the margin of the network model to be trained is greater than the margin of the original network model.

[0229] Again, in an embodiment of the present application, a method for calculating the loss value is provided. Through the above method, the loss value is calculated for the triplet image samples of two "goodcases" and one "badcase" in a targeted manner. Considering the annotation results of the new features, the target network model can support the metric representation of the newly added samples.

[0230] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in an optional embodiment provided by the embodiment of the present application, according to the labeling result corresponding to each image sample in the triple image sample, determining to adopt the target loss function may specifically include:

[0231] If two image samples in the triplet image sample are image samples with recognition errors, and one image sample is a correct sample with recognition errors, it is determined to adopt a third loss function, wherein the third loss function includes an original triplet loss function, a target triplet loss function, and a consistency loss function;

[0232] Based on the target embedding features corresponding to the sample image, the target loss function is used to determine the loss value corresponding to the triple image sample, which may specifically include:

[0233] Obtain the original embedded features corresponding to the sample image through the original network model;

[0234] Based on the original embedded features corresponding to the sample image, an original triplet loss function included in the third loss function is used to calculate the original triplet loss value;

[0235] Based on the target embedding features corresponding to the sample image, a target triplet loss function included in the third loss function is used to calculate the target triplet loss value;

[0236] Based on the target embedded features and the original embedded features corresponding to the image sample, a consistency loss value is calculated using a consistency loss function included in the third loss function;

[0237] The loss value corresponding to the triplet image sample is determined according to the original triplet loss value, the target triplet loss value and the consistency loss value.

[0238] In one or more embodiments, a method for calculating a loss value is introduced. It can be seen from the above embodiments that, in one case, the triplet image sample includes badcase, badcase and goodcase.

[0239] Specifically, for goodcase, we hope that the embeddings predicted by the original network model and the network model to be trained are as consistent as possible. At the same time, for the triplet loss function, it is necessary not only to satisfy the triplet loss function under the target embedding feature, but also to satisfy the triplet loss function under the image sample x. a Target embedding features, image samples x p The target embedding features and image samples x n The triple loss constraint under the original embedding feature of . Therefore, two triple loss functions are needed here, namely the target triple loss function loss tri-new And the original triple loss function loss tri-old Among them, the target triple loss function loss tri-new Indicates that all three image samples use the target embedding feature to calculate the metric learning margin of 0.8. The original triplet loss function loss tri-old Represents image sample x a and image sample x p The target embedding feature is used to calculate the metric learning margin of 0.6, and the image sample x n The original embedding features are used for calculation.

[0240] Based on this, the third loss function can be expressed as:

[0241] L3=loss tri-old0.6 +loss tri-new0.8 +α*loss con:n ;

[0242] Among them, L3 represents the loss value of the triplet image sample. tri-old0.6 Represents the original triplet loss function with a margin of 0.6. tri-new0.8 represents the target triplet loss function with a margin of 0.8. con:n represents the consistency loss function. α represents the weight value, for example, α = 0.5.

[0243] More specifically, as described in the above embodiment, the consistency loss function loss con:nIndicates that one image sample in the triple image sample needs to calculate KL-loss. Assuming that a triple image sample includes image sample A, the KL-loss of image sample A is calculated based on the original embedding features and target embedding features of image sample A. Thus, the consistency loss value loss can be obtained. con:n .

[0244] The target triplet loss function can be expressed as:

[0245] loss tri-new0.8 =max(||x anew -x pnew ||-||x anew -x nnew ||+β,0);

[0246] Among them, loss tri-new0.8 Represents the target triplet loss value. x anew Represents the image sample x in the triplet image sample a The corresponding target embedding feature. pnew Represents the image sample x that is a positive sample in the triplet image sample p The corresponding target embedding feature. nnew Represents the image sample x that is a negative sample in the triplet image sample n The corresponding target embedding feature. β represents margin, which can be set to 0.8.

[0247] The original triplet loss function can be expressed as:

[0248] loss tri-old0.6 =max(||x anew -x pnew ||-||x anew -x nold ||+β,0);

[0249] Among them, loss tri-new0.8 Represents the target triplet loss value. x anew Represents the image sample x in the triplet image sample a The corresponding target embedding feature. pnew Represents the image sample x that is a positive sample in the triplet image sample p The corresponding target embedding feature. nold Represents the image sample x that is a negative sample in the triplet image sample n The corresponding original embedding feature. β represents margin, which can be set to 0.6.

[0250] It should be noted that the weight value α can also be adjusted to 0.1, that is, to increase the proportion of the triplet loss function, which is not limited here. The margin of the original network model can be set not only to 0.6, but also to other values, such as 1. The margin of the network model to be trained can be set not only to 0.8, but also to other values, such as 1.6. Usually, the margin of the network model to be trained is greater than the margin of the original network model.

[0251] Again, in an embodiment of the present application, a method for calculating the loss value is provided. Through the above method, the loss value is calculated for a triplet of image samples of one "goodcase" and two "badcases" in a targeted manner. Taking into account the annotation results of the new features, the target network model can support the metric representation of the newly added samples.

[0252] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in an optional embodiment provided by the embodiment of the present application, according to the labeling result corresponding to each image sample in the triple image sample, determining to adopt the target loss function may specifically include:

[0253] If each image sample in the triplet image sample is an image sample with recognition error, determining to adopt a fourth loss function, wherein the fourth loss function includes a triplet loss function;

[0254] Based on the target embedding features corresponding to the sample image, the target loss function is used to determine the loss value corresponding to the triple image sample, which may specifically include:

[0255] Based on the target embedding features corresponding to the sample image, the loss value corresponding to the triplet image sample is calculated using the triplet loss function included in the fourth loss function.

[0256] In one or more embodiments, a method for calculating a loss value is introduced. It can be seen from the above embodiments that, in one case, the triplet image sample includes badcase, badcase, and badcase.

[0257] Specifically, if each image sample in the triplet image sample is a badcase, it is hoped that the target embedding features of these badcases meet the requirements of triplet-loss. When the original embedding features and the target embedding features are mixed to calculate triplet-loss in learning, the margin needs to be smaller than 0.8 set by the original network model (for example, set to 0.6). When the image samples all use the target embedding features to calculate triplet-loss, the margin can be set to 0.8. The difference between 0.6 and 0.8 allows some image samples to meet the requirements of triplet-loss when the original embedding features and the target embedding features are not completely consistent (that is, the consistency loss is not strictly 0). If it is not set smaller than 0.8, the model cannot converge.

[0258] Based on this, the fourth loss function can be expressed as:

[0259] L4=loss tri-new0.8 =max(||x anew -x pnew ||-||x anew -x nnew ||+β,0);

[0260] Among them, L4 represents the loss value of the triplet image sample. tri-new0.8 represents the triplet loss function. anew Represents the image sample x in the triplet image sample a The corresponding target embedding feature. pnew Represents the image sample x that is a positive sample in the triplet image sample p The corresponding target embedding feature. nnew Represents the image sample x that is a negative sample in the triplet image sample n The corresponding target embedding feature. β represents margin, which can be set to 0.8.

[0261] Again, in an embodiment of the present application, a method for calculating the loss value is provided. Through the above method, the loss value is calculated for the three "badcase" triplet image samples in a targeted manner, thereby improving the reliability and flexibility of the calculation.

[0262] In combination with the above introduction, the image recognition method in this application will be introduced below. Figure 8 , an embodiment of the image recognition method in the embodiment of the present application includes:

[0263] 210. Obtain an image to be queried;

[0264] In one or more embodiments, the image recognition device obtains an image to be queried, for example, a user uploads an image to be queried, or selects an image from the background as the image to be queried. The image to be queried can be used for image retrieval or image recognition, etc. Image recognition belongs to category-level recognition, which does not consider the specific instance of the object, but only considers the category of the object (for example, a person, dog, cat or bird, etc.) to perform recognition and give the category to which the object belongs.

[0265] It should be noted that the image recognition device can be deployed on a server or a terminal device, or can be deployed in a system consisting of a server and a terminal device.

[0266] 220. Based on the image to be queried, obtaining the embedded features to be queried through the target network model;

[0267] In one or more embodiments, the image recognition device inputs the image to be queried into the target network model, and the target network model outputs the embedding features to be queried, wherein the embedding features to be queried can be represented as a 128-dimensional vector.

[0268] It should be noted that the target network model is obtained by training using the method provided in the above embodiment, and the training process will not be described in detail here.

[0269] 230. Obtain a recall result of the image to be queried from the image library according to the embedded feature to be queried and the Q embedded features to be matched, wherein the recall result includes K images, and K and Q are both integers greater than or equal to 1;

[0270] In one or more embodiments, the image recognition device may respectively calculate the distances between the embedding feature to be queried and the Q embedding features to be matched, thereby obtaining Q feature distances, wherein the smaller the feature distance, the higher the similarity.

[0271] Specifically, taking the "brute force retrieval" scenario as an example, first, the corresponding embedded features to be matched can be extracted from the Q images in the image library in advance, that is, after the image is input into the original network model, the original network model outputs the embedded features to be matched corresponding to the image. Then, the correspondence between the image and the embedded features to be matched is stored. Based on this, after the feature distance is calculated, the feature distances can be arranged in ascending order. Therefore, the top K images can be selected from the corresponding images after the feature distance arrangement as the recall results. It can be understood that the images stored in the image library are seed images.

[0272] 240. Determine an image recognition result of the image to be queried according to the recall result.

[0273] In one or more embodiments, the image recognition device may determine the image recognition result of the image to be queried based on the K recalled images. For example, the image with the smallest feature distance with the image to be queried among the K images may be used as the image recognition result. Alternatively, at least one image with a feature distance with the image to be queried that is less than a feature distance threshold among the K images may be used as the image recognition result. The image recognition result may include not only the image but also the category of the image, for example, the category of image A is "puppy".

[0274] In an embodiment of the present application, a method for image recognition is provided. In the above manner, a target network model is used to extract embedding features to be queried of a to-be-queried image and used for image recognition. The training process of the target network model avoids re-features of a large amount of inventory in the application. When the model is fine-tuned and updated using limited new data, there is no need to update the inventory on a large scale, thus avoiding the need for more graphics processing unit (GPU) resources for forward calculations of multiple network layers. This is very economical for businesses with large inventories or that require frequent model updates. And with the help of a distributed inventory deployed on multiple machines, each machine only needs to update a small number of badcase image samples for its own inventory.

[0275] Optionally, in the above Figure 8 On the basis of the corresponding embodiments, in an optional embodiment provided by the embodiment of the present application, obtaining the recall result of the image to be queried from the image library according to the embedded feature to be queried and the Q embedded features to be matched may specifically include:

[0276] Obtaining a first feature distance between the embedded feature to be queried and each of the Q embedded features to be matched, to obtain Q first feature distances, wherein each embedded feature to be matched is a cluster center of a cluster;

[0277] Obtain at least one clustering result from the Q clustering results according to the Q first feature distances;

[0278] Obtaining a second feature distance between the embedded feature to be queried and each image in at least one clustering result;

[0279] According to the second feature distance between each image, K images in the image library are used as the recall results of the query image.

[0280] In one or more embodiments, a method of image bucket retrieval is introduced. One "bucket" corresponds to one clustering result. The image library may include P images, so the embedding features of each image can be extracted, and then the clusters of the embedding features can be trained. For example, 100,000 cluster centers are trained using the embedding features corresponding to 100 million images. Based on this, the cluster centers can be used as indexes for retrieval, and an association relationship between the index and the images in the image library can be established.

[0281] Specifically, the cluster center can be represented as the embedded feature to be matched. For the sake of illustration, please refer to Fig. 9 , Fig. 9 This is a schematic diagram of bucket-based retrieval in an embodiment of the present application. As shown in the figure, in image retrieval, the embedding feature to be queried is respectively calculated with each embedded feature to be matched to obtain Q first feature distances. Find the smallest L (L is an integer greater than or equal to 1) first feature distances from the Q first feature distances, and obtain L indexes. Based on this, L clustering results are obtained from the Q clustering results, and each clustering result includes at least one image. Each image in the L clustering results is used as a candidate image, and the distance between the embedding feature to be queried and each candidate image is calculated to obtain multiple second feature distances. Finally, the smallest K second feature distances are selected from these second feature distances, so that the K images corresponding to the K second feature distances can be used as the recall results of the image to be queried.

[0282] Secondly, in an embodiment of the present application, a method of image bucket retrieval is provided. Through the above method, bucket retrieval can improve image retrieval efficiency and save resources required for image matching.

[0283] Optionally, in the above Figure 8 On the basis of the corresponding embodiments, in an optional embodiment provided by the embodiment of the present application, the image library includes P images, where P is an integer greater than or equal to 1;

[0284] Before obtaining the recall result of the image to be queried from the image library according to the embedded features to be queried and the Q embedded features to be matched, the following may also be included:

[0285] According to the embedded features of the image, cluster the P images to obtain Q clustering results and the cluster center of each clustering result;

[0286] Each cluster center in the Q clusters is used as an embedded feature to be matched, and Q embedded features to be matched are obtained.

[0287] In one or more embodiments, a method for updating cluster centers is introduced. After the target network model is updated, images similar to the bad case can be selected from the image library, and their corresponding embedding features can be updated, and then the cluster centers and their indexes can be updated, thereby supporting optimized retrieval under partial update features.

[0288] Specifically, for ease of understanding, see Fig.10 , Fig.10 This is a schematic diagram of implementing the training and application of the network model in the embodiment of the present application. As shown in the figure, in combination with the above-mentioned embodiment, it can be known that the newly added image sample pair set is input into the original network model, and clustering is performed based on the embedding features output by the original network model to obtain the sample center of the badcase. Similarly, the original image sample pair set is input into the original network model, and based on the embedding features output by the original network model and the sample center of the badcase, the original image samples in the original image sample pair set are annotated to find the badcase and goodcase. Thus, the model training is performed.

[0289] During the application process, the embedding features of the badcase in the image library can be updated, and the embedding features of the goodcase do not need to be updated. For the images in the image library, the nearest images are found from the badcase image library. When the distance between the image in the image library and the nearest badcase image (the distance is calculated using the original embedding features) is less than the distance threshold (d_badcase_thr), the original embedding features of the image are updated to the target embedding features under the target network model. Therefore, only a small number of inventory samples need to be updated to avoid updating all samples.

[0290] Based on this, according to the embedding features corresponding to the P images, the P images are clustered, that is, new cluster centers are trained, and the new cluster centers (that is, the embedding features to be matched) are used to build the index. It can be understood that when there are relatively few bad cases in the image library, for example, less than 1 / 1000 of the image library, the cluster centers do not need to be increased, otherwise the number of cluster centers is set proportionally. For example, when the bad cases in the image library are 1 / 100 of the full image library, the number of cluster centers is also 1 / 100 of the original number of cluster centers.

[0291] Secondly, in the embodiment of the present application, a method for updating the cluster center is provided, through which the new cluster center is trained using the updated target network model, thereby improving the accuracy of the cluster center. In addition, only a small number of images in the image library need to be updated, thereby saving processing resources.

[0292] The model training device in this application is described in detail below. Fig.11 , Fig.11 This is a schematic diagram of an embodiment of a model training device in an embodiment of the present application. The model training device 30 includes:

[0293] An acquisition module 310 is used to acquire an original image sample pair set and a newly added image sample pair set, wherein the original image sample pair set includes at least two original image sample pairs used to train the original network model, and the newly added image sample pair set includes at least two newly added image sample pairs with recognition errors;

[0294] A labeling module 320 is used to obtain the labeling result of each original image sample pair in the original image sample pair set according to the newly added image sample pair set, and obtain an expanded image sample pair set, wherein the expanded image sample pair set includes expanded image sample pairs labeled as being incorrectly recognized and expanded image sample pairs labeled as being correctly recognized;

[0295] A construction module 330 is used to construct a triple image sample set according to the expanded image sample pair set and the newly added image sample pair set, wherein the triple image sample set includes at least one triple image sample, and the triple image sample includes at least two image samples marked as being misidentified or correctly identified;

[0296] The acquisition module 310 is further used to acquire the target embedding feature corresponding to each sample image in each triple image sample through the network model to be trained based on the triple image sample set, wherein the network model to be trained has the same network structure as the original network model, and the network parameters of the network model to be trained are consistent with the network parameters of the original network model;

[0297] The training module 340 is used to update the model parameters of the network model to be trained for each triple image sample according to the annotation results corresponding to each image sample and the target embedded features corresponding to each sample image until the model training conditions are met to obtain the target network model.

[0298] In an embodiment of the present application, a model training device is provided, and the above-mentioned device is used to annotate the original image sample pair set using the newly added image sample pair set, so as to obtain the identification of goodcase and badcase. Since goodcase can make the feature distribution consistent, and badcase can perform effective metric learning, it is possible to achieve the effect of optimizing badcase while keeping the features close to goodcase. Therefore, these image sample pairs (i.e., goodcase and badcase) are used to update the model, without re-extracting all image samples to extract embedded features, thereby saving computing resources and improving the efficiency of model updating.

[0299] Optionally, in the above Fig.11 On the basis of the corresponding embodiment, in another embodiment of the model training device 30 provided in the embodiment of the present application,

[0300] The labeling module 320 is specifically used to obtain a first embedding feature corresponding to each original image sample through the original network model based on the original image sample pair set, wherein the original image sample pair set includes 2M original image samples, and M is an integer greater than 1;

[0301] Based on the set of newly added image sample pairs, obtaining a second embedding feature corresponding to each newly added image sample through the original network model, wherein the set of newly added image sample pairs includes 2N newly added image samples, where N is an integer greater than 1;

[0302] Performing clustering processing on the 2N newly added image samples according to the second embedded feature corresponding to each newly added image sample to obtain at least two cluster centers, wherein each cluster center corresponds to a preset pattern of recognition errors;

[0303] According to at least two cluster centers, the shortest distance corresponding to each newly added image sample in each newly added image sample pair is obtained to obtain 2N shortest distances;

[0304] According to at least two cluster centers, obtaining the distance between the first embedded feature corresponding to each original image sample and each cluster center;

[0305] According to the relationship between the distance between the first embedded feature corresponding to each original image sample and each cluster center and the distance threshold, each original image sample pair in the original image sample pair set is labeled to obtain an expanded image sample pair set, wherein the distance threshold is determined based on 2N shortest distances.

[0306] In an embodiment of the present application, a model training device is provided. By using the above device, each original image sample pair can be automatically labeled using a newly added image sample pair set without manual labeling, thereby saving manual labeling costs and improving labeling efficiency.

[0307] Optionally, in the above Fig.11 On the basis of the corresponding embodiment, in another embodiment of the model training device 30 provided in the embodiment of the present application,

[0308] The marking module 320 is specifically configured to use the maximum value among the 2N shortest distances as a distance threshold;

[0309] If the distance between the first embedded feature and the cluster center is less than the distance threshold, the original image sample corresponding to the first embedded feature is marked as an incorrectly recognized expanded image sample;

[0310] If the distance between the first embedded feature and the cluster center is greater than or equal to the distance threshold, the original image sample corresponding to the first embedded feature is marked as a correctly identified expanded image sample;

[0311] Two expanded image samples corresponding to the same original image sample pair are regarded as an expanded image sample pair until a set of expanded image sample pairs is obtained.

[0312] In an embodiment of the present application, a model training device is provided. By using the above device, a distance threshold can be used to determine whether the original image sample is a bad case. The maximum value of the shortest distance is used as the distance threshold. The distance threshold can be maximized under reasonable circumstances, thereby screening out more expanded image sample pairs marked as bad cases.

[0313] Optionally, in the above Fig.11 On the basis of the corresponding embodiment, in another embodiment of the model training device 30 provided in the embodiment of the present application,

[0314] The labeling module 320 is specifically configured to label the other expanded image sample as an incorrectly recognized expanded image sample if one expanded image sample corresponding to the same original image sample pair is labeled as an incorrectly recognized expanded image sample and the other expanded image sample is labeled as an correctly recognized expanded image sample.

[0315] In an embodiment of the present application, a model training device is provided. The above device is used to automatically correct the annotation results, which not only improves the accuracy of the annotation results, but also increases the number of bad cases to a certain extent, thereby achieving the purpose of maximum training and avoiding sample pair mismatch during model training.

[0316] Optionally, in the above Fig.11On the basis of the corresponding embodiment, in another embodiment of the model training device 30 provided in the embodiment of the present application,

[0317] A construction module 330 is specifically used to obtain similar image sample pairs from the expanded image sample pair set and the newly added image sample pair set;

[0318] Eliminate similar image sample pairs from the expanded image sample pair set and the newly added image sample pair set to obtain (M+N-1) image sample pairs, wherein the expanded image sample pair set includes M expanded image sample pairs, and the newly added image sample pair set includes N newly added image sample pairs, wherein M and N are both integers greater than 1;

[0319] Select one image sample from each of the (M+N-1) image sample pairs to obtain (M+N-1) image samples;

[0320] Calculate the distance between each image sample in the (M+N-1) image samples and the similar image sample pair to obtain (M+N-1) sample distances;

[0321] Obtain the first T sample distances with the smallest sample distance from the (M+N-1) sample distances, where T is an integer greater than or equal to 1;

[0322] The image samples corresponding to each sample distance in the T sample distances are respectively combined with similar image sample pairs to obtain T triple image samples, wherein the T triple image samples are included in the triple image sample set.

[0323] In an embodiment of the present application, a model training device is provided, and the device is used to merge the expanded image sample pair set and the newly added image sample pair set into a training set, thereby achieving the purpose of expanding the badcase data set. In addition, triple image samples are constructed for subsequent model training to improve the reliability of model training.

[0324] Optionally, in the above Fig.11 On the basis of the corresponding embodiment, in another embodiment of the model training device 30 provided in the embodiment of the present application,

[0325] The training module 340 is specifically used to determine, for each triple image sample, the target loss function to be used according to the labeling result corresponding to each image sample in the triple image sample;

[0326] For each triple image sample, based on the target embedding features corresponding to the sample image, the target loss function is used to determine the loss value corresponding to the triple image sample;

[0327] According to the loss value corresponding to each triplet image sample, the overall loss value is determined;

[0328] The overall loss value is adopted; the model parameters of the network model to be trained are updated.

[0329] In an embodiment of the present application, a model training device is provided. The above device is used to perform triple learning using the bad cases that are reasonably mined during the model training phase. The features of the good cases in the database are not updated. During the learning process, the distribution of the new features of the good cases (i.e., the embedding of the expanded image samples) is made the same as the distribution of the old features (i.e., the embedding of the original image samples). Thus, in the application, better measurement effects can be obtained without updating the good cases. For the bad cases in the database, their features are updated while satisfying the measurement effects between them and the old features of the good cases.

[0330] Optionally, in the above Fig.11 On the basis of the corresponding embodiment, in another embodiment of the model training device 30 provided in the embodiment of the present application,

[0331] The training module 340 is specifically used to determine to use a first loss function if each image sample in the triplet image sample is a correctly identified image sample, wherein the first loss function includes a triplet loss function and a consistency loss function;

[0332] The training module 340 is specifically used to obtain the original embedded features corresponding to the sample image through the original network model;

[0333] Based on the target embedding feature corresponding to the sample image, a triplet loss function included in the first loss function is used to calculate a triplet loss value;

[0334] Based on the target embedded features and the original embedded features corresponding to the sample image, a consistency loss value is calculated using a consistency loss function included in the first loss function;

[0335] The loss value corresponding to the triplet image sample is determined according to the triplet loss value and the consistency loss value.

[0336] In an embodiment of the present application, a model training device is provided. The above device is used to specifically calculate the loss value of three "goodcase" triplet image samples, thereby improving the reliability and flexibility of the calculation.

[0337] Optionally, in the above Fig.11 On the basis of the corresponding embodiment, in another embodiment of the model training device 30 provided in the embodiment of the present application,

[0338] The training module 340 is specifically used to determine to use a second loss function if two image samples in the triple image sample are correctly identified image samples and one image sample is an incorrectly identified image sample, wherein the second loss function includes an original triple loss function, a target triple loss function and a consistency loss function;

[0339] The training module 340 is specifically used to obtain the original embedded features corresponding to the sample image through the original network model;

[0340] Based on the original embedded features corresponding to the sample image, the original triplet loss function included in the second loss function is used to calculate the original triplet loss value;

[0341] Based on the target embedding features corresponding to the sample image, a target triplet loss function included in the second loss function is used to calculate the target triplet loss value;

[0342] Based on the target embedded features and the original embedded features corresponding to the image sample, a consistency loss value is calculated using a consistency loss function included in the second loss function;

[0343] The loss value corresponding to the triplet image sample is determined according to the original triplet loss value, the target triplet loss value and the consistency loss value.

[0344] In an embodiment of the present application, a model training device is provided. The above device is used to specifically calculate the loss value of a triplet of image samples of two "goodcases" and one "badcase". Taking into account the annotation results of the new features, the target network model can support the metric representation of the newly added samples.

[0345] Optionally, in the above Fig.11 On the basis of the corresponding embodiment, in another embodiment of the model training device 30 provided in the embodiment of the present application,

[0346] The training module 340 is specifically used to determine to use a third loss function if two image samples in the triple image sample are image samples with recognition errors, and one image sample is a correct sample with recognition errors, wherein the third loss function includes an original triple loss function, a target triple loss function, and a consistency loss function;

[0347] The training module 340 is specifically used to obtain the original embedded features corresponding to the sample image through the original network model;

[0348] Based on the original embedded features corresponding to the sample image, an original triplet loss function included in the third loss function is used to calculate the original triplet loss value;

[0349] Based on the target embedding features corresponding to the sample image, a target triplet loss function included in the third loss function is used to calculate the target triplet loss value;

[0350] Based on the target embedded features and the original embedded features corresponding to the image sample, a consistency loss value is calculated using a consistency loss function included in the third loss function;

[0351] The loss value corresponding to the triplet image sample is determined according to the original triplet loss value, the target triplet loss value and the consistency loss value.

[0352] In an embodiment of the present application, a model training device is provided. The above device is used to specifically calculate the loss value of a triplet image sample of one "goodcase" and two "badcases". Taking into account the annotation results of the new features, the target network model can support the metric representation of the newly added samples.

[0353] Optionally, in the above Fig.11 On the basis of the corresponding embodiment, in another embodiment of the model training device 30 provided in the embodiment of the present application,

[0354] The training module 340 is specifically configured to determine to use a fourth loss function if each image sample in the triplet image sample is an image sample with a recognition error, wherein the fourth loss function includes a triplet loss function;

[0355] The training module 340 is specifically used to calculate the loss value corresponding to the triplet image sample based on the target embedding feature corresponding to the sample image using the triplet loss function included in the fourth loss function.

[0356] In an embodiment of the present application, a model training device is provided. Using the above device, loss values ​​are calculated for three "badcase" triplet image samples in a targeted manner, thereby improving the reliability and flexibility of the calculation.

[0357] The image recognition device in this application is described in detail below. Fig.12 , Fig.12 This is a schematic diagram of an embodiment of an image recognition device in an embodiment of the present application. The image recognition device 40 includes:

[0358] An acquisition module 410 is used to acquire an image to be queried;

[0359] The acquisition module 410 is further used to acquire the embedded features to be queried through the target network model based on the image to be queried, wherein the target network model is trained by using the training method of any one of claims 1 to 10 above;

[0360] The acquisition module 410 is further used to acquire a recall result of the image to be queried from the image library according to the embedded feature to be queried and the Q embedded features to be matched, wherein the recall result includes K images, and K and Q are both integers greater than or equal to 1;

[0361] The determination module 420 is used to determine the image recognition result of the image to be queried according to the recall result.

[0362] In an embodiment of the present application, an image recognition device is provided. The above device is used to extract the embedding features to be queried of the image to be queried using the target network model, and used for image recognition. The training process of the target network model avoids the need to re-extract features for a large amount of inventory in the application. When the model is fine-tuned and updated using limited new data, there is no need to update the inventory on a large scale, avoiding the need for more graphics processing unit (GPU) resources for forward calculations of multiple network layers. This is very economical for businesses with large inventories or that require frequent model updates. And with the help of a distributed inventory deployed on multiple machines, each machine only needs to update a small number of badcase image samples for its own inventory.

[0363] Optionally, in the above Fig.12 On the basis of the corresponding embodiment, in another embodiment of the image recognition device 40 provided in the embodiment of the present application,

[0364] The acquisition module 410 is specifically used to acquire a first feature distance between the embedded feature to be queried and each of the embedded features to be matched in the Q embedded features to be matched, and obtain Q first feature distances, wherein each embedded feature to be matched is a cluster center of a cluster;

[0365] Obtain at least one clustering result from the Q clustering results according to the Q first feature distances;

[0366] Obtaining a second feature distance between the embedded feature to be queried and each image in at least one clustering result;

[0367] According to the second feature distance between each image, K images in the image library are used as the recall results of the query image.

[0368] In an embodiment of the present application, an image recognition device is provided. By using the above device and bucket retrieval, the image retrieval efficiency can be improved and the resources required for image matching can be saved.

[0369] Fig.13: is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 500 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 522 (for example, one or more processors) and a memory 532, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 542 or data 544. Among them, the memory 532 and the storage medium 530 can be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the computer device. Furthermore, the central processing unit 522 can be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the computer device 500.

[0370] The computer device 500 may also include one or more power supplies 526, one or more wired or wireless network interfaces 550, one or more input and output interfaces 558, and / or one or more operating systems 541, such as Windows Server 2003, ... TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0371] The steps performed by the computer device in the above embodiment can be based on the Fig.13 The computer device structure shown.

[0372] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.

[0373] The present application also provides a computer program product including a program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.

[0374] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0375] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0376] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0377] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0378] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0379] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A network model training method, characterized in that: include: Obtaining an original image sample pair set and a newly added image sample pair set, wherein the original image sample pair set includes at least two original image sample pairs used to train the original network model, and the newly added image sample pair set includes at least two newly added image sample pairs with recognition errors; Based on the original image sample pair set, obtaining a first embedding feature corresponding to each original image sample through the original network model, wherein the original image sample pair set includes 2M original image samples, and M is an integer greater than 1; Based on the set of newly added image sample pairs, obtaining a second embedding feature corresponding to each newly added image sample through the original network model, wherein the set of newly added image sample pairs includes 2N newly added image samples, where N is an integer greater than 1; According to the second embedded feature corresponding to each of the newly added image samples, clustering processing is performed on the 2N newly added image samples to obtain at least two cluster centers, wherein each cluster center corresponds to a preset pattern of recognition errors; According to the at least two cluster centers, the shortest distance corresponding to each newly added image sample in each newly added image sample pair is obtained to obtain 2N shortest distances; According to the at least two cluster centers, obtaining the distance between the first embedded feature corresponding to each original image sample and each cluster center; According to the relationship between the distance between the first embedded feature corresponding to each original image sample and each cluster center and the distance threshold, each original image sample pair in the original image sample pair set is labeled to obtain an extended image sample pair set, specifically including: taking the maximum value of the 2N shortest distances as the distance threshold; if the distance between the first embedded feature and the cluster center is less than the distance threshold, the original image sample corresponding to the first embedded feature is labeled as an incorrectly recognized extended image sample; if the distance between the first embedded feature and the cluster center is greater than or equal to the distance threshold, the original image sample corresponding to the first embedded feature is labeled as an correctly recognized extended image sample; two extended image samples corresponding to the same original image sample pair are regarded as an extended image sample pair until the extended image sample pair set is obtained, wherein the extended image sample pair set includes extended image sample pairs labeled as incorrectly recognized and extended image sample pairs labeled as correctly recognized; Constructing a triple image sample set according to the expanded image sample pair set and the newly added image sample pair set, wherein the triple image sample set includes at least one triple image sample, and the triple image sample includes at least two image samples marked as being misidentified or correctly identified; Based on the triple image sample set, obtaining the target embedding feature corresponding to each sample image in each triple image sample through the network model to be trained, wherein the network model to be trained has the same network structure as the original network model, and the network parameters of the network model to be trained are consistent with the network parameters of the original network model; For each triplet image sample, the model parameters of the network model to be trained are updated according to the annotation results corresponding to each image sample and the target embedding features corresponding to each sample image, until the model training conditions are met to obtain the target network model.

2. The training method according to claim 1, characterized in that: The step of taking two expanded image samples corresponding to the same original image sample pair as an expanded image sample pair comprises: If one augmented image sample corresponding to the same original image sample pair is marked as an incorrectly recognized augmented image sample, and another augmented image sample is marked as a correctly recognized augmented image sample, the other augmented image sample is marked as an incorrectly recognized augmented image sample.

3. The training method according to claim 1, characterized in that: The step of constructing a triplet image sample set according to the expanded image sample pair set and the newly added image sample pair set includes: Acquire similar image sample pairs from the expanded image sample pair set and the newly added image sample pair set; Eliminating the similar image sample pairs from the expanded image sample pair set and the newly added image sample pair set to obtain (M+N-1) image sample pairs, wherein the expanded image sample pair set includes M expanded image sample pairs, and the newly added image sample pair set includes N newly added image sample pairs, wherein both M and N are integers greater than 1; Select one image sample from each of the (M+N-1) image sample pairs to obtain (M+N-1) image samples; Calculating the distance between each image sample in the (M+N-1) image samples and the similar image sample pair to obtain (M+N-1) sample distances; Obtain the first T sample distances with the smallest sample distance from the (M+N-1) sample distances, where T is an integer greater than or equal to 1; The image samples corresponding to each sample distance in the T sample distances are respectively combined with the similar image sample pairs to obtain T triple image samples, wherein the T triple image samples are included in the triple image sample set.

4. The training method according to claim 1, characterized in that: The updating of the model parameters of the network model to be trained according to the annotation results corresponding to each image sample and the target embedding features corresponding to each sample image for each triplet image sample includes: For each triplet of image samples, determining to use a target loss function according to a labeling result corresponding to each image sample in the triplet of image samples; For each triplet image sample, based on the target embedding feature corresponding to the sample image, the target loss function is used to determine the loss value corresponding to the triplet image sample; According to the loss value corresponding to each triplet image sample, the overall loss value is determined; The overall loss value is adopted; and the model parameters of the network model to be trained are updated.

5. The training method according to claim 4, characterized in that: The step of determining the target loss function to be used according to the labeling result corresponding to each image sample in the triplet image sample includes: If each image sample in the triplet of image samples is an image sample that is correctly identified, determining to use a first loss function, wherein the first loss function includes a triplet loss function and a consistency loss function; The method of determining the loss value corresponding to the triplet image sample by using the target loss function based on the target embedding feature corresponding to the sample image includes: Obtaining original embedded features corresponding to the sample image through the original network model; Based on the target embedding feature corresponding to the sample image, a ternary loss value is calculated using the ternary loss function included in the first loss function; Based on the target embedded features and the original embedded features corresponding to the sample image, a consistency loss value is calculated using a consistency loss function included in the first loss function; A loss value corresponding to the triplet image sample is determined according to the triplet loss value and the consistency loss value.

6. The training method according to claim 4, characterized in that: The step of determining the target loss function to be used according to the labeling result corresponding to each image sample in the triplet image sample includes: If two image samples in the triplet image sample are correctly identified image samples, and one image sample is an incorrectly identified image sample, determining to use a second loss function, wherein the second loss function includes an original triplet loss function, a target triplet loss function, and a consistency loss function; The method of determining the loss value corresponding to the triplet image sample by using the target loss function based on the target embedding feature corresponding to the sample image includes: Obtaining original embedded features corresponding to the sample image through the original network model; Based on the original embedded features corresponding to the sample image, an original triplet loss function included in the second loss function is used to calculate an original triplet loss value; Based on the target embedding feature corresponding to the sample image, a target triplet loss function included in the second loss function is used to calculate a target triplet loss value; Based on the target embedded features and the original embedded features corresponding to the image sample, a consistency loss value is calculated using a consistency loss function included in the second loss function; The loss value corresponding to the triplet image sample is determined according to the original triplet loss value, the target triplet loss value and the consistency loss value.

7. The training method according to claim 4, characterized in that: The step of determining the target loss function to be used according to the labeling result corresponding to each image sample in the triplet image sample includes: If two image samples in the triplet image sample are image samples with recognition errors, and one image sample is a correct sample with recognition errors, it is determined to adopt a third loss function, wherein the third loss function includes an original triplet loss function, a target triplet loss function, and a consistency loss function; The method of determining the loss value corresponding to the triplet image sample by using the target loss function based on the target embedding feature corresponding to the sample image includes: Obtaining original embedded features corresponding to the sample image through the original network model; Based on the original embedded features corresponding to the sample image, an original triplet loss function included in the third loss function is used to calculate an original triplet loss value; Based on the target embedding feature corresponding to the sample image, a target triplet loss function included in the third loss function is used to calculate a target triplet loss value; Based on the target embedded features and the original embedded features corresponding to the image sample, a consistency loss value is calculated using a consistency loss function included in the third loss function; The loss value corresponding to the triplet image sample is determined according to the original triplet loss value, the target triplet loss value and the consistency loss value.

8. The training method according to claim 4, characterized in that: The step of determining the target loss function to be used according to the labeling result corresponding to each image sample in the triplet image sample includes: If each image sample in the triplet of image samples is an image sample with recognition error, determining to use a fourth loss function, wherein the fourth loss function includes a triplet loss function; The method of determining the loss value corresponding to the triplet image sample by using the target loss function based on the target embedding feature corresponding to the sample image includes: Based on the target embedding features corresponding to the sample image, the loss value corresponding to the triplet image sample is calculated using the triplet loss function included in the fourth loss function.

9. A method for image recognition, characterized in that: include: Get the image to be queried; Based on the image to be queried, acquiring the embedded features to be queried through a target network model, wherein the target network model is trained by using the training method described in any one of claims 1 to 8; According to the embedded features to be queried and the Q embedded features to be matched, a recall result of the image to be queried is obtained from an image library, wherein the recall result includes K images, and both K and Q are integers greater than or equal to 1; According to the recall result, an image recognition result of the image to be queried is determined.

10. The method according to claim 9, characterized in that The step of obtaining a recall result of the image to be queried from an image library according to the embedded feature to be queried and the Q embedded features to be matched includes: Obtaining a first feature distance between the embedded feature to be queried and each of the Q embedded features to be matched, to obtain Q first feature distances, wherein each embedded feature to be matched is a cluster center of a cluster; Obtaining at least one clustering result from the Q clustering results according to the Q first feature distances; Obtaining a second feature distance between the embedded feature to be queried and each image in the at least one clustering result; According to the second characteristic distance between each of the images, the K images in the image library are used as the recall results of the image to be queried.

11. A model training device, characterized in that: include: An acquisition module, used to acquire an original image sample pair set and a newly added image sample pair set, wherein the original image sample pair set includes at least two original image sample pairs used to train the original network model, and the newly added image sample pair set includes at least two newly added image sample pairs with recognition errors; A labeling module, configured to obtain a first embedding feature corresponding to each original image sample through the original network model based on the original image sample pair set, wherein the original image sample pair set includes 2M original image samples, and M is an integer greater than 1; The labeling module is further used to obtain a second embedding feature corresponding to each newly added image sample through the original network model based on the newly added image sample pair set, wherein the newly added image sample pair set includes 2N newly added image samples, and N is an integer greater than 1; The labeling module is further used to perform clustering processing on the 2N newly added image samples according to the second embedded feature corresponding to each newly added image sample to obtain at least two cluster centers, wherein each cluster center corresponds to a preset pattern of recognition errors; The labeling module is further used to obtain the shortest distance corresponding to each newly added image sample in each newly added image sample pair according to the at least two cluster centers, and obtain 2N shortest distances; The labeling module is further used to obtain the distance between the first embedded feature corresponding to each original image sample and each cluster center according to the at least two cluster centers; The labeling module is further used to label each original image sample pair in the original image sample pair set according to the relationship between the distance between the first embedded feature corresponding to each original image sample and each cluster center and the distance threshold, so as to obtain the extended image sample pair set, specifically including: taking the maximum value of the 2N shortest distances as the distance threshold; if the distance between the first embedded feature and the cluster center is less than the distance threshold, labeling the original image sample corresponding to the first embedded feature as an incorrectly recognized extended image sample; if the distance between the first embedded feature and the cluster center is greater than or equal to the distance threshold, labeling the original image sample corresponding to the first embedded feature as an correctly recognized extended image sample; taking two extended image samples corresponding to the same original image sample pair as an extended image sample pair until the extended image sample pair set is obtained, wherein the extended image sample pair set includes extended image sample pairs labeled as incorrectly recognized and extended image sample pairs labeled as correctly recognized; A construction module, configured to construct a triple image sample set according to the expanded image sample pair set and the newly added image sample pair set, wherein the triple image sample set includes at least one triple image sample, and the triple image sample includes at least two image samples marked as being misidentified or correctly identified; The acquisition module is further used to acquire the target embedding feature corresponding to each sample image in each triple image sample through the network model to be trained based on the triple image sample set, wherein the network model to be trained has the same network structure as the original network model, and the network parameters of the network model to be trained are consistent with the network parameters of the original network model; The training module is used to update the model parameters of the network model to be trained for each triple image sample according to the annotation results corresponding to each image sample and the target embedded features corresponding to each sample image until the model training conditions are met to obtain the target network model.

12. An image recognition device, characterized in that: include: An acquisition module, used to acquire the image to be queried; The acquisition module is further used to acquire the embedded features to be queried through the target network model based on the image to be queried, wherein the target network model is trained by using the training method described in any one of claims 1 to 8; The acquisition module is further used to acquire a recall result of the image to be queried from an image library according to the embedded feature to be queried and the Q embedded features to be matched, wherein the recall result includes K images, and both K and Q are integers greater than or equal to 1; A determination module is used to determine the image recognition result of the image to be queried according to the recall result.

13. A computer device, characterized in that: include: Memory, processor, and bus system; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, and the processor is used to execute the method according to any one of claims 1 to 8 according to the instructions in the program code, or execute the method according to any one of claims 9 to 10; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

14. A computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute the training method according to any one of claims 1 to 8, or the method according to any one of claims 9 to 10.

15. A computer program product comprising a computer program and instructions, characterized in that: When the computer program / instructions are executed by a processor, the training method according to any one of claims 1 to 8 is implemented, or the method according to any one of claims 9 to 10 is executed.

Images