Feature extraction model processing, sample retrieval method, device and computer equipment
By using the noise loss and classification loss training process in the feature extraction model, the target feature extraction model is generated, which solves the problem of low accuracy of traditional feature extraction models under noise labels, and improves the accuracy of feature extraction and the generalization ability of the model .
Patent Information
- Application Number
- CN202210007882.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-05
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-01-05
AI Technical Summary
Traditional feature extraction models often find it difficult to obtain high-accuracy features during training, especially in the presence of noise labels, which leads to insufficient generalization capabilities of the model.
By extracting the features of the training samples, determining the candidate tag set based on the classification results and category labels, selecting representative sample features that match the characteristics of the training samples, and training the feature extraction model based on the noise loss and classification loss, and generating the target feature extraction model to improve the Accuracy of feature extraction.
It improves the accuracy of the feature extraction model, can generalize and learn in the presence of noise samples, enhances the generalization ability of the model, and avoids the negative impact of noise samples on the model.
Smart Images

Figure CN114358188B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a feature extraction model processing method, apparatus, computer equipment, storage medium and computer program product, as well as a sample retrieval method, apparatus, computer equipment, storage medium and computer program product. Background Art
[0002] With the development of computer technology, machine learning technology has emerged. Through machine learning, machine learning models for various tasks can be trained. For example, a feature extraction model for feature extraction tasks can be trained. The feature extraction model can extract the features corresponding to the input sample, and the input sample can be identified based on the features to obtain the recognition results. For example, a sentence can be feature extracted to obtain a feature vector representing the sentence, and the sentence can be translated based on the feature vector.
[0003] In traditional technology, feature extraction models can be trained through training samples. However, it is often the case that the features extracted by the trained feature extraction models have relatively low accuracy. Summary of the invention
[0004] Based on this, it is necessary to provide a feature extraction model processing method, device, computer equipment, storage medium and computer program product, as well as a sample retrieval method, device, computer equipment, storage medium and computer program product to address the above technical problems.
[0005] On the one hand, the present application provides a feature extraction model processing method. The method includes: extracting training sample features of training samples through a feature extraction model to be trained; the training samples have corresponding category labels; classifying the training samples based on the training sample features to obtain classification results, and determining classification losses based on the classification results and the category labels; determining a candidate label set corresponding to the training samples, and obtaining representative sample features corresponding to each candidate label in the candidate label set; selecting representative sample features that match the training sample features from each of the representative sample features, and determining the selected representative sample features as target representative sample features; determining the candidate labels to which the target representative sample features belong from the candidate label set, and determining the candidate labels to which the target representative sample features belong as recommended labels for the training samples; when there is a difference between the recommended labels and the category labels, determining noise losses based on the classification results and the recommended labels; training the feature extraction model based on the noise losses and the classification losses, and obtaining a target feature extraction model when a training stop condition is met; the target feature extraction model is used to extract sample features of input samples.
[0006] On the other hand, the present application also provides a feature extraction model processing device. The device comprises: obtaining a query sample and a candidate recall sample set; inputting the query sample and the candidate recall sample in the candidate recall sample set into a target feature extraction model to obtain a query sample feature corresponding to the query sample and a candidate recall sample feature corresponding to the candidate recall sample; determining a target retrieval sample corresponding to the query sample from the candidate recall sample set based on the query sample feature and the candidate recall sample feature; wherein the target feature extraction model is obtained by training a feature extraction model to be trained according to noise loss and classification loss; the classification loss value is determined based on a classification result and a category label of a training sample; the classification result is obtained by classifying the training sample based on the training sample feature; the training sample feature is obtained by extracting features from the training sample by the feature extraction model; the noise loss is determined based on the classification result and the recommended label when there is a difference between the recommended label and the category label; the recommended label is a candidate label to which the target representative sample feature determined from the candidate label set belongs; the target representative sample feature matches the training sample feature and is selected from the representative sample features corresponding to each candidate label in the candidate label set; the candidate label set corresponds to the training sample.
[0007] On the other hand, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the feature extraction model processing method when executing the computer program.
[0008] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the feature extraction model processing method are implemented.
[0009] On the other hand, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the feature extraction model processing method are implemented.
[0010] The feature extraction model processing method, device, computer equipment, storage medium and computer program product extract training sample features of training samples through the feature extraction model to be trained, wherein the training samples have corresponding category labels, classify the training samples based on the training sample features, obtain classification results, determine the classification loss based on the classification results and the category labels, determine the candidate label set corresponding to the training samples, and obtain the representative sample features corresponding to each candidate label in the candidate label set, select the representative sample features that match the training sample features from each representative sample feature, determine the selected representative sample features as the target representative sample features, and determine the candidate label set to which the target representative sample features belong from the candidate label set. Select labels and determine the candidate labels to which the target representative sample features belong as the recommended labels for the training samples. The recommended labels are possible correct labels for the training samples. Therefore, when there is a difference between the recommended labels and the category labels, the training samples may be noise samples. Since the noise conditions of the recommended labels and the original category labels cannot be absolutely determined, the feature extraction model can be further trained based on the noise loss and classification loss. When the training stop conditions are met, the target feature extraction model is obtained, so that the feature extraction model can use a noise-friendly method to perform generalized learning based on the recommended labels, so that the processing of noise samples produces positive benefits for the model, thereby improving the accuracy of the features extracted by the feature extraction model.
[0011] On the other hand, the present application provides a sample retrieval method. The method comprises: obtaining a query sample and a candidate recall sample set; inputting the query sample and the candidate recall sample in the candidate recall sample set into a target feature extraction model to obtain a query sample feature corresponding to the query sample and a candidate recall sample feature corresponding to the candidate recall sample; wherein the target feature extraction model is obtained by training a feature extraction model to be trained according to noise loss and classification loss; the classification loss value is determined based on a classification result and a category label of a training sample; the classification result is obtained by classifying the training sample based on the training sample feature; the training sample feature is obtained by extracting features from the training sample by the feature extraction model; the noise loss is determined based on the classification result and the recommended label when there is a difference between the recommended label and the category label; the recommended label is a candidate label to which the target representative sample feature determined from the candidate label set belongs; the target representative sample feature matches the training sample feature and is selected from the representative sample features corresponding to each candidate label in the candidate label set; the candidate label set corresponds to the training sample and determines the target retrieval sample corresponding to the query sample from the candidate recall sample set based on the query sample feature and the candidate recall sample feature.
[0012] On the other hand, the present application provides a sample retrieval device. The device includes: a sample acquisition module, which is used to acquire a query sample and a candidate recall sample set; a feature extraction module, which is used to input the query sample and the candidate recall sample in the candidate recall sample set into a target feature extraction model to obtain the query sample feature corresponding to the query sample and the candidate recall sample feature corresponding to the candidate recall sample; a retrieval module, which is used to determine the target retrieval sample corresponding to the query sample from the candidate recall sample set based on the query sample feature and the candidate recall sample feature; wherein the target feature extraction model is obtained by training the feature extraction model to be trained according to the noise loss and the classification loss; the classification loss value is based on the classification result and the training The classification result is obtained by classifying the training sample based on the training sample features; the training sample features are obtained by extracting features of the training sample through the feature extraction model; the noise loss is determined based on the classification result and the recommended label when there is a difference between the recommended label and the category label; the recommended label is a candidate label to which the target representative sample feature determined from the candidate label set belongs; the target representative sample feature matches the training sample feature and is selected from the representative sample features corresponding to each candidate label in the candidate label set; the candidate label set corresponds to the training sample.
[0013] On the other hand, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above sample retrieval method are implemented.
[0014] On the other hand, the present application further provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above-mentioned sample retrieval method when executed by a processor.
[0015] On the other hand, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above sample retrieval method are implemented.
[0016] The above-mentioned sample retrieval method, device, computer equipment, storage medium and computer program product, since the target feature extraction model is obtained by training the feature extraction model to be trained based on noise loss and classification loss; the classification loss value is determined based on the classification result and the category label of the training sample; the classification result is obtained by classifying the training sample based on the training sample feature; the training sample feature is obtained by extracting the feature of the training sample by the feature extraction model; the noise loss is determined based on the classification result and the recommended label when there is a difference between the recommended label and the category label; the recommended label is the candidate label to which the target representative sample feature determined from the candidate label set belongs; the target representative sample feature matches the training sample feature, and is selected from the representative sample features corresponding to each candidate label in the candidate label set, and the candidate label set corresponds to the training sample, therefore, the target feature extraction model is obtained by generalization learning based on the recommended label using a noise-friendly method, and the accuracy of the features extracted by the target feature extraction model is high, thereby improving the accuracy of sample retrieval. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 An application environment diagram of a feature extraction model processing method in an embodiment;
[0018] Figure 2 A schematic diagram of a process flow of a feature extraction model processing method in one embodiment;
[0019] Figure 3 is a schematic diagram of representative sample features selected in one embodiment;
[0020] Figure 4 A schematic diagram of a process flow of a feature extraction model processing method in another embodiment;
[0021] Figure 5 A schematic diagram of a process flow of a feature extraction model processing method in yet another embodiment;
[0022] Figure 6 Schematic diagram of a training process of a feature extraction model in one embodiment;
[0023] Figure 7 is a flow chart of a sample retrieval method in one embodiment;
[0024] Figure 8 is a structural block diagram of a feature extraction model processing device in one embodiment;
[0025] Fig. 9 is a structural block diagram of a sample retrieval device in one embodiment;
[0026] Fig.10 is an internal structure diagram of a computer device in one embodiment;
[0027] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0029] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0030] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.
[0031] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, automatic driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.
[0032] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.
[0033] The solution provided in the embodiments of the present application involves artificial intelligence computer vision, machine learning and other technologies, which are specifically described by the following embodiments:
[0034] The feature extraction model processing method and sample retrieval method provided in this application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 can be, but is not limited to, a laptop, a smart phone, a tablet computer, a desktop computer, a smart TV, a car terminal, and a portable wearable device. The terminal can be provided with an application, through which the input sample can be retrieved to obtain the target retrieval sample. The application can refer to a client installed in the terminal. The client (also referred to as an application client, APP client) refers to a program installed and running in the terminal; the application can also refer to an installation-free application, that is, an application that can be used without downloading and installing. This type of application is also commonly known as a mini-program, which usually runs in the client as a subroutine; the application can also refer to a web application opened through a browser; and so on. The server 104 can be implemented with an independent server or a server cluster or cloud server consisting of multiple servers.
[0035] Both the terminal 102 and the server 104 can be used independently to execute the feature extraction model processing and sample retrieval method provided in the embodiments of the present application.
[0036] For example, the server 104 can extract training sample features of the training samples through the feature extraction model to be trained, wherein the training samples have corresponding category labels, classify the training samples based on the training sample features to obtain classification results, determine the classification loss based on the classification results and the category labels, determine the candidate label set corresponding to the training samples, and obtain the representative sample features corresponding to each candidate label in the candidate label set, select representative sample features that match the training sample features from each representative sample feature, determine the selected representative sample features as the target representative sample features, determine the candidate labels to which the target representative sample features belong from the candidate label set, and determine the candidate labels to which the target representative sample features belong as the recommended labels for the training samples, and when there is a difference between the recommended labels and the category labels, determine the noise loss based on the classification results and the recommended labels, train the feature extraction model based on the noise loss and the classification loss, and obtain the target feature extraction model when the training stop condition is met.
[0037] Server 104 can obtain a query sample and a candidate recall sample set, input the query sample and the candidate recall sample in the candidate recall sample set into a target feature extraction model, obtain query sample features corresponding to the query sample and candidate recall sample features corresponding to the candidate recall sample, and determine a target retrieval sample corresponding to the query sample from the candidate recall sample set based on the query sample features and the candidate recall sample features.
[0038] The terminal 102 and the server 104 can also be used together to execute the feature extraction model processing and sample retrieval method provided in the embodiments of the present application.
[0039] For example, the server 104 may obtain a training sample from the terminal. After training the target feature extraction model, the server 104 may send the trained target feature extraction model to the terminal, and the terminal extracts the sample features from the input sample based on the target feature extraction model. The terminal may further retrieve the target retrieval sample corresponding to the input sample from the database of the server based on the extracted sample features.
[0040] In one embodiment, Figure 2 As shown, a feature extraction model processing method is provided, and the method is described by taking the method executed by a computer device as an example. It can be understood that the computer device can be Figure 1 The terminal 102 shown may also be a server 104. In this embodiment, the feature extraction model processing method includes the following steps:
[0041] Step 202: extracting training sample features of the training samples through the feature extraction model to be trained; the training samples have corresponding category labels.
[0042] Among them, the training sample refers to the content sample used to train the feature extraction model, and the content can be any one of text, audio or image. There is a corresponding category label for the training sample, and the category label corresponding to the training sample is used to identify the category of the training sample. For example, if the training sample is an image, the category label corresponding to the training sample can be animal species such as dogs, cats, fish, plant species such as corals, pine trees, osmanthus, or object types such as magnifying glasses, cabinets, and water bottles. The category label corresponding to the training sample can be used as supervision information for the feature extraction model to be trained. The category label corresponding to the training sample can be one or more, and multiple means at least two. When there are multiple category labels for the training sample, it means that the training sample can be classified into multiple different categories. Taking the training sample as a picture as an example, if the training sample contains both cats and dogs, then the training sample can be classified into the cat category or the dog category.
[0043] The category labels of training samples are marked in advance. When the marked labels are wrong or missing, the training samples are noise samples. There are two types of noise samples, namely, label missing noise samples and label error noise samples. When a sample contains multiple labels, but only one is marked or not all of these labels are marked, it is called label missing. Such samples with incomplete labels are called label missing noise samples. Label error noise refers to the noise generated by inconsistent sample-label information, such as (blue sky image, ocean) forming label noise. At this time, the image (sky image) does not have a label (ocean). Learning machine learning models under noise samples is called weakly supervised learning. At this time, noise samples need to be processed, otherwise model bias will occur, resulting in learning failure, poor results and other problems.
[0044] The feature extraction model to be trained refers to the feature extraction model that needs to adjust parameters. The feature extraction model is a machine learning model used to extract features and output feature vectors. The feature extraction model includes at least an embedding model. The embedding model is used to output feature vectors. The feature vector output by the embedding model can be called embedding. The embedding model can be a model composed of one or more fully connected layers. The output features of the embedding model are the features extracted by the feature extraction model.
[0045] In one embodiment, the feature vector output by the embedding model can be normalized so that each component of the feature vector has a value range of -1 to 1. In other embodiments, the feature vector can also be binary quantized to obtain binary quantized features. Binary quantization refers to the process of binary encoding the feature. For example, the feature can be encoded as a binary code with values of 0 and 1. Bit compression can also be performed during the encoding process. For example, the feature vector can be compressed to 48 bits. The feature obtained by binary quantization of the feature vector can be called a hash feature. In this case, the embedding model can be called a hash quantization model.
[0046] In one embodiment, the feature extraction model may only include an embedding model, and the input end of the embedding model may be connected to the output end of the trained basic neural network model, and receive the output of the basic neural network model as input. The basic neural network model may be a model for extracting feature information contained in the content, and the basic neural network model may be a neural network based on artificial intelligence, such as a convolutional neural network (CNN), or a network such as ResNet101 (deep residual network 101) or ResNet18 (deep residual network 18). The convolutional neural network is a type of feedforward neural network (Feedforward Neural Networks) that includes convolution calculations and has a deep structure. Specifically, after obtaining the training sample, the computer device inputs the training sample into the trained basic neural network model, extracts feature information through the trained basic neural network model, and inputs the extracted feature information into the embedding model to be trained to obtain a feature vector, which is the training sample feature corresponding to the training sample. For example, taking the training sample as an image, the image can be input into the trained CNN model to extract the deep features of the image, and the extracted deep features can be input into the embedding model to obtain the feature vector of the image.
[0047] In other embodiments, the feature extraction model may include a basic neural network model and an embedding model, in which case the basic neural network model and the embedding model are trained together as feature extraction models. Specifically, after acquiring the training sample, the computer device inputs the training sample into the basic neural network model to be trained, extracts feature information through the basic neural network model to be trained, and inputs the extracted feature information into the embedding model to be trained to obtain a feature vector, which is the training sample feature corresponding to the training sample.
[0048] It is understandable that different feature extraction models can be trained for different types of samples. For example, if the sample is an image type sample, then an image feature extraction model can be trained; if the sample is a speech type sample, then a speech feature extraction model can be trained.
[0049] In one embodiment, a computer device locally stores a set of labeled training samples. The computer device can obtain training samples from the locally stored training sample set, input the training samples into a feature extraction model to be trained, extract features through the feature extraction model, and output sample features corresponding to the training samples to obtain training sample features of the training samples. It can be understood that for the same training sample, different training sample features can be obtained under different feature extraction models. For example, the training sample feature can be a D-dimensional feature vector directly output by the embedding model, or a feature obtained by normalizing the feature vector, or a hash feature.
[0050] In one embodiment, the training samples in the training sample set can be labeled by searching from an Internet search engine. For example, taking the training samples as images, specifically, for a certain label, the label can be entered in the search engine, and the first N images returned are taken as images with the label. For example, if a search engine searches for golden retrievers, the first 500 images returned are saved as image samples with the label "golden retriever" in the training sample set.
[0051] Step 204: classify the training samples based on the training sample features to obtain classification results, and determine the classification loss based on the classification results and the category labels.
[0052] Among them, classification refers to the classification of training samples based on their features to determine the specific category to which the training samples belong. The classification result is used to characterize the category information of the specific category to which the training samples belong. The classification loss is used to characterize the difference between the classification result and the category label. The classification loss is positively correlated with the difference, that is, the larger the difference, the larger the classification loss.
[0053] Specifically, the computer device can classify the training samples based on the training sample features to obtain a classification result, and then calculate the classification loss based on the difference between the classification result and the category label.
[0054] In one embodiment, the computer device can use a classification model to classify the training sample features, that is, input the training sample features into the classification model, and obtain the output of the classification model as the classification result. Among them, the classification model refers to a machine learning model that can perform category recognition. In some embodiments, the classification model may include one or more fully connected layers. When the classification model includes multiple fully connected layers, the first few fully connected layers can extract semantic features from the training sample features, so it can be called a semantic layer Semantic_layer, and the last fully connected layer is used for classification, which is called a classification layer. The number of semantic layers can be determined as needed, and the semantic layer can be deepened for situations where more feature intersections are required. By setting the semantic layer, high-order features can be abstracted to more fully mine classification information.
[0055] In one embodiment, the classification result may be a category identifier used to characterize the category to which the training sample belongs. For example, if the classification model classifies N categories, the first category may be represented by an N-dimensional vector (1, 0, 0, 0, ..., 0), and the second category may be represented by a vector (0, 1, 0, 0, ..., 0), and so on. In another embodiment, the classification result may be a probability used to characterize the category to which the training sample belongs. For example, if the classification model can be used to classify N categories, the classification model may output an N-dimensional probability vector, wherein the probability of each dimension is used to characterize the likelihood that the training sample belongs to the category.
[0056] In other embodiments, the computer device performs classification based on the training sample features, which may be to calculate the feature distance between the training sample features and the category features corresponding to each candidate category. When the feature distance between the training sample features and the category features is less than the distance threshold, the training category corresponding to the category features is determined as the classification category to which the current training sample belongs. At this time, the current classification result may be the category identifier of the classification category. The category features corresponding to the candidate categories may be determined in the following manner: collect multiple content samples corresponding to the candidate categories, extract the feature vectors of these content samples through the embedding model, and then calculate an average vector as the category feature corresponding to the candidate category.
[0057] Step 206: determine a candidate label set corresponding to the training sample, and obtain representative sample features corresponding to each candidate label in the candidate label set.
[0058] Among them, the candidate label set corresponding to the training sample refers to the label set where the correct category label of the training sample is located. Each label in the candidate label set can be used as a candidate label. The category identified by each candidate label is the candidate category when classifying the training sample. The representative sample feature corresponding to the candidate label is the sample feature that can represent the candidate label. The representative sample feature corresponding to the candidate label can be, for example, the sample feature of a sample under the category identified by the candidate label. The representative sample feature corresponding to the candidate label can be one or more. The representative sample feature corresponding to the candidate label can also be called the prototype of the label.
[0059] Specifically, after the computer device determines the candidate label set corresponding to the training sample, for each candidate label in the candidate label set, the computer device can obtain the representative sample feature corresponding to the candidate label, and the computer device can establish an association relationship between the candidate label and the corresponding representative sample feature.
[0060] In one embodiment, the computer device may determine labels corresponding to all candidate categories in the process of classifying the training samples as candidate labels corresponding to the training samples, and obtain a candidate label set.
[0061] In one embodiment, for each candidate tag in the candidate tag set, the computer device may cluster multiple samples under the category identified by the candidate tag, and determine the representative sample feature of the candidate tag based on the clustering result. For example, the computer device may determine the cluster center in each cluster cluster obtained by clustering as the representative sample feature of the candidate tag.
[0062] Step 208: Select a representative sample feature that matches the training sample feature from each representative sample feature, and determine the selected representative sample feature as the target representative sample feature.
[0063] Among them, the matching of the representative sample feature and the training sample feature means that the similarity between the representative sample feature and the training sample feature satisfies the similarity condition or the difference between the representative sample feature and the training sample feature satisfies the difference condition. The similarity condition can be, for example, that the similarity between the representative sample feature and the training sample feature is greater than a preset similarity threshold or is ranked before a preset ranking threshold. The similarity can be, for example, cosine similarity. The difference condition can be, for example, that the feature distance between the representative sample feature and the training sample feature is less than a preset distance threshold or is ranked before a preset ranking threshold. The feature distance can be, for example, L2 distance. The target representative sample feature can be one or more.
[0064] Specifically, after obtaining the representative sample features corresponding to each candidate label, the computer device can select representative sample features that match the training sample features from these representative sample features, and determine the selected representative sample features as target representative sample features.
[0065] In one embodiment, the computer device may calculate the feature distances between the training sample features and each representative sample feature, and determine the representative sample features whose feature distances are less than a preset distance threshold as target representative sample features. The preset distance threshold may be set as required.
[0066] In another embodiment, the computer device may respectively calculate the feature distances between the training sample features and each representative sample feature, sort the feature distances, determine one or more smaller feature distances according to the sorting results, and determine the representative sample features corresponding to these feature distances as the target representative sample features. For example, the computer device may determine the smallest feature distance according to the sorting results, and determine the representative sample feature corresponding to the feature distance as the target representative sample feature.
[0067] Step 210, determining the candidate tag to which the target representative sample feature belongs from the candidate tag set, and determining the candidate tag to which the target representative sample feature belongs as the recommended tag for the training sample.
[0068] The candidate label to which the target representative sample feature belongs refers to the candidate label corresponding to the target representative sample feature. The recommended label refers to the possible correct category label of the training sample.
[0069] Specifically, the computer device can determine the candidate label that has an association relationship with the target representative sample feature as the candidate label belonging to the target representative sample feature based on the pre-established association relationship between the candidate label and the representative sample feature, and determine the candidate label belonging to the target representative sample feature as the recommended label for the training sample.
[0070] Step 212 : when there is a difference between the recommended label and the category label, determine the noise loss based on the classification result and the recommended label.
[0071] Step 214, training the feature extraction model according to the noise loss and the classification loss, and obtaining the target feature extraction model when the training stop condition is met; the target feature extraction model is used to extract the sample features of the input sample.
[0072] The difference between the recommended label and the category label refers to the mismatch between the recommended label and the category label. For example, the recommended label is cat, while the category label is dog. Another example is that the recommended labels include cat and dog, while the category label only includes dog. Noise loss is used to characterize the difference between the classification result and the recommended label. The greater the difference, the greater the noise loss.
[0073] Specifically, after the computer device compares the recommended label with the category label, it determines that there is a difference between the recommended label and the category label, indicating that the training sample at this time may have an incorrect label or a missing label, that is, the training sample may be a noise sample. Since the noise of the originally labeled category label and the recommended label cannot be absolutely determined, the computer device can perform generalization loss learning and learn the category label and the recommended label at the same time, that is, calculate the noise loss based on the difference between the classification result and the recommended label, and then calculate the target loss based on the noise loss and the classification loss. Finally, adjust the parameters of the feature extraction model according to the target loss and continue training until the training stop condition is met, and obtain the target feature extraction model. The trained target feature extraction model can be used to extract features from the input sample to obtain the sample features of the training sample. It can be understood that the input sample and the training sample are samples of the same type. For example, when the training sample is an image, the target feature extraction model can extract features from the input image to obtain the image features of the input image.
[0074] In one embodiment, when the computer device calculates the target loss based on the noise loss and the classification loss, it can be calculated in a regularized manner, that is, the noise loss is weighted by a first weight to obtain a first weighted result, and the classification loss is weighted by a second weight to obtain a second weighted result, and the first weighted result and the second weighted result are statistically calculated to obtain the target loss. The first weight and the second weight can be pre-set according to prior knowledge, and the first weight can be, for example, 0.5, and the second weight can be, for example, 0.5. The statistics can be, for example, summing or averaging.
[0075] In one embodiment, during the training process, the computer device may use any one of the stochastic gradient descent algorithm, Adagrad (Adaptive Gradient) algorithm, Adadelta (an improvement of the AdaGrad algorithm), RMSprop (an improvement of the AdaGrad algorithm), Adam (Adaptive Moment Estimation) algorithm, etc. to adjust the model parameters of the feature extraction model.
[0076] In one embodiment, the training stop condition may be that the model parameters no longer change, the loss reaches the minimum value, or the number of training times reaches the maximum number of iterations, etc. In other embodiments, the computer device may perform multiple rounds (epochs) of iterative training on the feature extraction model, and after each round of training, the target loss of that round is obtained. The training stop condition may be that the average target loss in a certain epoch no longer decreases.
[0077] In the feature extraction model processing method, the training sample features of the training samples are extracted by the feature extraction model to be trained, wherein the training samples have corresponding category labels, the training samples are classified based on the training sample features, and the classification results are obtained, the classification loss is determined based on the classification results and the category labels, the candidate label set corresponding to the training samples is determined, and the representative sample features corresponding to each candidate label in the candidate label set are obtained, and the representative sample features matching the training sample features are selected from the representative sample features, and the selected representative sample features are determined as the target representative sample features, and the candidate labels to which the target representative sample features belong are determined from the candidate label set, and the target representative sample features are assigned to the target representative sample features. The candidate label to which this feature belongs is determined as the recommended label of the training sample. The recommended label is the possible correct label of the training sample. Therefore, when there is a difference between the recommended label and the category label, the training sample may be a noise sample. Since the noise situation of the recommended label and the original category label cannot be absolutely determined, the feature extraction model can be further trained according to the noise loss and classification loss. When the training stop condition is met, the target feature extraction model is obtained, so that the feature extraction model can use a noise-friendly method to perform generalization learning according to the recommended label, so that the processing of noise samples has a positive benefit to the model, thereby improving the accuracy of the features extracted by the feature extraction model.
[0078] Furthermore, since there is no need to introduce additional noise prior knowledge or manual labeling, noise samples can be determined, avoiding additional manpower investment and improving the training efficiency of the feature extraction model.
[0079] In one embodiment, representative sample features corresponding to each candidate tag in a candidate tag set are obtained, including: obtaining a candidate sample set corresponding to each candidate tag; for each candidate tag, performing feature extraction on each candidate sample in the candidate sample set corresponding to the candidate tag to obtain each candidate sample feature corresponding to the candidate tag; clustering each candidate sample feature corresponding to the candidate tag, and determining the representative sample feature corresponding to the candidate tag based on the clustering result.
[0080] The candidate sample set corresponding to the candidate label refers to a set consisting of multiple training samples under the category identified by the candidate label. For example, if the candidate label is dog and the sample is an image, multiple images under the dog category can be selected to form the candidate sample set. The clustering result refers to multiple clusters obtained by clustering, and each cluster has a corresponding cluster center. The candidate sample feature refers to the feature obtained by extracting the candidate sample.
[0081] Specifically, after the computer device obtains the candidate sample sets corresponding to each candidate label, for each candidate label, the computer device can extract features of each candidate sample in the candidate sample set corresponding to the candidate label to obtain candidate sample features of each candidate sample.
[0082] In one embodiment, the computer device can input each candidate sample into the feature extraction model to obtain the candidate sample features of each candidate sample. Since the candidate sample features are obtained through the feature extraction model, it is equivalent to jointly training the feature extraction model and the noise model, which can avoid the adverse effects of the noise model alone on the feature extraction model due to poor judgment results of some samples during the iteration process.
[0083] The computer device can further cluster each candidate sample feature to obtain multiple clusters corresponding to the candidate label, and the computer device can further determine the representative sample feature corresponding to the candidate label based on each cluster. The computer device can cluster each candidate sample feature using a clustering method in traditional technology, for example, a k-means method can be used for clustering.
[0084] In one embodiment, the computer device may select the cluster center of the cluster as the representative sample feature corresponding to the candidate tag. In other embodiments, the computer device may also select the candidate sample feature closer to the cluster center as the representative sample feature. Figure 3 FIG. 1 is a schematic diagram of representative sample features selected in an embodiment, referring to FIG. Figure 3 The clustering unit clusters the features of each candidate sample to obtain three clusters, and the cluster centers of the clusters are Figure 3The candidate sample features represented by the triangle in the figure and the candidate sample features represented by the circle connected to the triangle are in the same cluster as the candidate sample features represented by the triangle. Prototypes can be selected from each cluster. These prototypes together constitute a multi-prototype representation. The multi-prototype representation is to represent the candidate label set through multiple prototypes. These prototypes can represent the candidate label set. Taking the cluster where triangle 302 is located as an example, the four candidate sample features in the dotted circle can be used as representative sample features of the candidate label (i.e., four prototypes). The triangle in the dotted circle indicates that it is not only a prototype, but also the cluster center of the cluster where it is located. The circle in the dotted circle represents the prototype of other non-cluster centers. The non-cluster center prototype plays an important role in the representation of rare samples under this label. In one embodiment, taking into account the problem of unbalanced learning tasks under large-scale data, such as in multi-label learning, some labels appear less frequently in training samples, such as rare auroras, resulting in few aurora-related training samples. The computer device can adopt different feature modeling methods. Specifically, for a candidate label, if the number of training samples corresponding to the candidate label is less than a preset threshold, a first number of training samples are selected to form a candidate sample set corresponding to the candidate label. If the number of training samples corresponding to the candidate label is greater than the preset threshold, a second number of training samples are selected to form a candidate sample set corresponding to the candidate label, wherein the second number is greater than the first number.
[0085] In the above embodiment, by clustering the candidate sample features corresponding to the candidate labels, and determining the representative sample features corresponding to the candidate labels based on the clustering results, the obtained representative sample features are more accurate.
[0086] In one embodiment, each candidate sample feature corresponding to a candidate label is clustered, and a representative sample feature corresponding to the candidate label is determined based on the clustering result, including: clustering each candidate sample feature to obtain multiple clusters; each cluster has a corresponding cluster center; for each cluster, from the candidate sample features corresponding to the cluster, a candidate sample feature whose feature distance with the cluster center meets a first distance condition; and the selected candidate sample feature is determined as the representative sample feature corresponding to the candidate label.
[0087] The characteristic distance meeting the first distance condition may be that the characteristic distance is less than a preset distance threshold, or the ranking of the characteristic distance is before the preset ranking threshold. The characteristic distance may be an L2 distance.
[0088] Specifically, for each candidate label, the computer device clusters each candidate sample feature corresponding to the candidate label to obtain multiple clusters. For each cluster, the computer device calculates the feature distance between each candidate sample feature in the cluster and the cluster center, and selects the candidate sample feature whose feature distance with the cluster center meets the first distance condition based on the calculated feature distance as the representative sample feature of the candidate label. Since each candidate label corresponds to multiple clusters, multiple representative sample features can be obtained for each candidate label.
[0089] In one embodiment, the computer device may compare each feature distance with a preset feature distance threshold, and if the feature distance is less than the preset distance threshold, the candidate sample feature corresponding to the feature distance may be determined as the representative sample feature corresponding to the candidate label. The preset distance threshold may be set as needed.
[0090] In another embodiment, the computer device may sort the feature distances to select the feature distance closest to the cluster center, and determine the candidate sample feature corresponding to the feature distance as the representative sample feature corresponding to the candidate label. For example, the feature distances may be sorted in ascending order, the feature distance ranked first may be selected, and the candidate sample feature corresponding to the feature distance may be determined as the representative sample feature corresponding to the candidate label.
[0091] In the above embodiment, for each candidate label, multiple clusters are obtained by clustering the candidate sample features corresponding to the candidate label, and candidate sample features whose feature distances with the cluster centers satisfy the first distance condition are selected from the multiple clusters, so that representative sample features can be quickly selected.
[0092] In one embodiment, Figure 4 As shown, a feature extraction model processing method is provided, and the method is described by taking the method executed by a computer device as an example. It can be understood that the computer device can be Figure 1 The terminal 102 shown may also be a server 104. In this embodiment, the feature extraction model processing method includes the following steps:
[0093] Step 402: extracting training sample features of the training samples through the feature extraction model to be trained; the training samples have corresponding category labels.
[0094] Step 404: classify the training samples based on the training sample features to obtain classification results, and determine the classification loss based on the classification results and the category labels.
[0095] Step 406: determine the candidate label set corresponding to the training sample, and obtain the candidate sample set corresponding to each candidate label in the candidate label set.
[0096] Step 408: for each candidate label, input each candidate sample in the candidate sample set corresponding to the candidate label into the feature extraction model respectively, perform feature extraction on each candidate sample based on the feature extraction model, and obtain the candidate sample features corresponding to each candidate sample.
[0097] Step 410 , clustering the candidate sample features corresponding to the candidate labels, and determining the representative sample features corresponding to the candidate labels based on the clustering results.
[0098] Step 412: Select a representative sample feature that matches the training sample feature from each representative sample feature, and determine the selected representative sample feature as the target representative sample feature.
[0099] Step 414, determine the candidate label to which the target representative sample feature belongs from the candidate label set, and determine the candidate label to which the target representative sample feature belongs as the recommended label for the training sample.
[0100] Step 416 , when there is a difference between the recommended label and the category label, determine the noise loss based on the classification result and the recommended label.
[0101] Step 418, according to the noise loss and the classification loss, the feature extraction model is trained in the current round to obtain an intermediate feature extraction model.
[0102] Among them, one round of training refers to a complete training of the feature extraction model using all training samples in the training sample set.
[0103] Specifically, the computer device may obtain noise loss for all training samples in the training sample set where the training sample is located according to the above steps 402 to 416, and then perform the current round of training on the feature extraction model based on the noise loss and classification loss of each sample to obtain an intermediate feature extraction model.
[0104] Step 420, determine whether the training stop condition is met, if not, proceed to step 422; if yes, proceed to step 424.
[0105] Among them, the training stop condition can be that the average target loss in a certain epoch no longer decreases.
[0106] Specifically, when each round of training is completed, the computer device will obtain the target loss of that round. The computer device can further calculate the evaluation target loss of each completed round of training. When the average target loss no longer decreases in a certain epoch, it is determined that the training stop condition is met.
[0107] In step 422 , the intermediate feature extraction model is used as the feature extraction model to be trained, and the process proceeds to step 402 .
[0108] Step 424, using the intermediate feature extraction model as the target feature extraction model.
[0109] The target feature extraction model is used to extract sample features of the input sample. The extracted sample features can be used to identify the input sample, and the extracted sample features can also be used to retrieve similar samples from the sample database for deduplication or recommendation.
[0110] It should be noted that the above steps 402-404 are Figure 2 Steps 202 to 204 are consistent with those in Figure 2 The steps 208 to 212 in the embodiment are consistent with each other, and the specific explanation of these steps can refer to Figure 2 The description of the steps in the present application will not be repeated here.
[0111] In the above embodiment, the candidate sample features of each candidate sample in the candidate sample set of the candidate label are extracted by the feature extraction model, and the representative sample features are obtained by clustering. Since the recommended label can be determined according to the matching relationship between the representative sample feature and the training sample feature, the recommended label and the category label can be compared to determine whether it is a noise sample. Therefore, the feature extraction model is equivalent to being used to constitute a noise judgment model. In this embodiment, it is equivalent to jointly training the feature extraction model and the noise judgment model. In the process of multiple rounds of iterative training, the recommended labels become more and more accurate, and the noise judgment results become more and more accurate. The judgment results of the noise model for label recommendation of samples in multiple rounds of training iterations are applied to the feature extraction model to avoid the influence of a single noise judgment on the learning effect. The generalization learning task of the feature extraction model is generated with the help of the noise judgment results to avoid the adverse effects of poor judgment results of some samples on the feature extraction model during the iteration of the noise model.
[0112] In one embodiment, representative sample features that match training sample features are selected from each representative sample feature, including: obtaining feature distances between the training sample features and each representative sample feature; selecting representative sample features whose feature distances satisfy a second distance condition from each representative sample feature to obtain target representative sample features; the second distance condition includes at least one of the feature distance being less than a preset distance threshold or the feature distance being ranked before a preset ranking threshold.
[0113] In one embodiment, the computer device calculates the feature distance between the training sample feature and each representative sample feature, obtains a preset distance threshold, and determines the representative sample feature whose feature distance is less than the preset distance threshold as the representative sample feature matching the training sample feature.
[0114] In another embodiment, the computer device calculates the feature distances between the training sample feature and each representative sample feature, sorts each feature distance, selects representative sample features corresponding to each of a preset number of feature distances sorted before a preset sorting threshold, and determines the selected representative sample features as representative sample features that match the training sample feature. For example, the computer device may sort the feature distances in ascending order, select a feature distance from the feature distances sorted before the third place, and determine the representative sample feature corresponding to the feature distance as the representative sample feature that matches the training sample feature.
[0115] In the above embodiment, by obtaining the feature distance between the training sample feature and each representative sample feature, the representative sample feature matching the training sample feature is selected based on the feature distance, so that the matching representative sample feature can be obtained quickly and accurately.
[0116] In one embodiment, from each representative sample feature, a representative sample feature whose feature distance satisfies a second distance condition is selected to obtain a target representative sample feature, including: sorting each feature distance from small to large to select a representative sample feature corresponding to the minimum feature distance to obtain a first target representative sample feature, and selecting a representative sample feature corresponding to the second minimum feature distance to obtain a second target representative sample feature; determining a candidate label to which the target representative sample feature belongs from a candidate label set, and determining the candidate label to which the target representative sample feature belongs as a recommended label for the training sample, including: determining a candidate label to which the first target representative sample feature belongs from the candidate label set to obtain a first recommended label, and determining a candidate label to which the second representative sample feature belongs to obtain a second recommended label.
[0117] Specifically, the computer device can sort the feature distances from small to large, determine the feature distance ranked first as the minimum feature distance, determine the representative sample feature corresponding to the minimum feature distance as the first representative sample feature, determine the feature distance ranked second as the second minimum feature distance, determine the representative sample feature corresponding to the second minimum feature distance as the second representative sample feature, and determine both the first representative sample feature and the second representative sample feature as representative sample features that match the training sample features, that is, obtain two target representative sample features. Since there are two target representative sample features, when the two target representative sample features are different, the computer device can obtain two recommended labels. Since it is impossible to absolutely determine the noise situation of the recommended label and the original category label, the computer device can use the generalization loss learning method to learn the two recommended labels and the original category label as learning targets at the same time, thereby improving the generalization performance of the feature extraction model for noise samples.
[0118] In one embodiment, Figure 5 As shown, a feature extraction model processing method is provided, and the method is described by taking the method executed by a computer device as an example. It can be understood that the computer device can be Figure 1 The terminal 102 shown may also be a server 104. In this embodiment, the feature extraction model processing method includes the following steps:
[0119] Step 502: extracting training sample features of training samples through the feature extraction model to be trained; the training samples have corresponding category labels.
[0120] Step 504: classify the training samples based on the training sample features to obtain classification results, and determine the classification loss based on the classification results and the category labels.
[0121] Step 506: determine a candidate label set corresponding to the training sample, and obtain representative sample features corresponding to each candidate label in the candidate label set.
[0122] Step 508, obtain the feature distance between the training sample feature and each representative sample feature, sort the feature distances from small to large, select the representative sample feature corresponding to the minimum feature distance to obtain the first target representative sample feature, and select the representative sample feature corresponding to the second minimum feature distance to obtain the second target representative sample feature.
[0123] Step 510: From the candidate tag set, determine the candidate tag to which the first target representative sample feature belongs to obtain a first recommended tag, and determine the candidate tag to which the second representative sample feature belongs to obtain a second recommended tag.
[0124] Step 512, when there is a difference between the first recommended label and the category label and there is a difference between the second recommended label and the category label, a first loss is determined based on the difference between the first recommended label and the classification result, and a second loss is determined based on the difference between the second recommended label and the classification result, and the first loss and the second loss are statistically calculated to obtain a noise loss.
[0125] Among them, the first loss is used to characterize the difference between the first recommended label and the classification result, and the first loss is positively correlated with the difference between the first recommended label and the classification result, that is, the greater the difference, the greater the first loss. The second loss is used to characterize the difference between the second recommended label and the classification result, and the second loss is positively correlated with the difference between the second recommended label and the classification result, that is, the greater the difference, the greater the second loss.
[0126] Specifically, the computer device can use a regularized method to count the first loss and the second loss, that is, weight the first loss according to the first noise loss weight to obtain the first weighted loss, and weight the first loss according to the second loss weight to obtain the second weighted loss, and count the first weighted loss and the second weighted loss to obtain the noise loss. The first loss weight and the second loss weight can be pre-set according to prior knowledge, and the first loss weight can be, for example, 0.3, and the second loss weight can be, for example, 0.3. Statistics can be, for example, summing or averaging.
[0127] Step 514, training the feature extraction model according to the noise loss and the classification loss, and obtaining the target feature extraction model when the training stop condition is met; the target feature extraction model is used to extract the sample features of the input sample.
[0128] It should be noted that the above steps 502-506 are Figure 2 Steps 202 to 206 are consistent with step 514. Figure 2 The specific explanation of these steps can refer to the step 214 in Figure 2 The description of the steps in the present application will not be repeated here.
[0129] In the above embodiment, the computer device can obtain two recommended labels by selecting the representative sample features corresponding to the minimum feature distance to obtain the first target representative sample features, and selecting the representative sample features corresponding to the second minimum feature distance to obtain the second target representative sample features. Training the samples is equivalent to learning three labels at the same time, further improving the generalization performance of the model against noise.
[0130] In one embodiment, when there is a difference between the recommended label and the category label, the noise loss is determined based on the classification result and the recommended label, including: comparing the recommended label and the category label to obtain a comparison result; when the comparison result indicates that the recommended label and the category label are inconsistent, the noise loss is determined based on the classification result and the recommended label; the above method also includes: when the comparison result indicates that the recommended label and the category label are consistent, the target loss is calculated based on the classification result and the category label; the feature extraction model is trained based on the target loss, and when the training stop condition is met, the target feature extraction model is obtained.
[0131] Specifically, the computer device compares the recommended label and the category label. When the comparison result indicates that the recommended label and the category label are inconsistent, it means that there is a difference between the recommended label and the category label, that is, the recommended label and the category label do not match. At this time, the training sample may be a noise sample, and the computer device can determine the noise loss based on the classification result and the recommended label. When the comparison result indicates that the recommended label and the category label are consistent, it means that the recommended label and the category label match, that is, the recommended labels are all original labels. At this time, the training sample may be a non-noise sample. The computer device can directly calculate the target loss based on the classification result and the category label, train the feature extraction model based on the target loss, and obtain the target feature extraction model when the training stop condition is met.
[0132] In the above embodiment, by comparing the recommended label and the category label, the computer device can adaptively select the loss calculation method based on the comparison result. When it is not necessary to calculate the noise loss, the target loss training model calculated based on the classification result and the category label is used to improve the accuracy and flexibility of the training process.
[0133] In one embodiment, a feature extraction model is trained according to noise loss and classification loss, and a target feature extraction model is obtained when a training stop condition is met, including: obtaining a comparison sample extraction feature corresponding to a comparison training sample of the training sample, and obtaining a feature extraction loss based on the difference between the training sample feature and the comparison sample extraction feature; obtaining a target loss based on noise loss, feature extraction loss and classification loss; adjusting parameters of the feature extraction model to be trained based on the target loss and continuing training, and obtaining a target feature extraction model when the training stop condition is met.
[0134] The comparison training sample refers to a content sample used for comparison with the training sample to determine the feature extraction loss. The comparison training sample may include at least one of a positive comparison training sample or a negative comparison training sample, wherein the positive comparison training sample refers to a content sample similar to the training sample, and the negative comparison training sample refers to a content sample that is not similar to the current training sample.
[0135] Specifically, the computer device can perform feature extraction on the comparison training samples based on the feature extraction model to be trained, obtain the sample extraction features corresponding to the comparison training samples as the comparison sample extraction features, obtain the feature extraction loss based on the difference between the training sample features and the comparison sample extraction features, perform weighted summation of the noise loss, feature extraction loss and classification loss to obtain the target loss, adjust the parameters of the feature extraction model to be trained based on the target loss and continue training, and obtain the target feature extraction model when the training stop condition is met.
[0136] In one embodiment, the computer device obtains the feature extraction loss based on the difference between the training sample feature and the comparison sample extracted feature by: calculating the cosine similarity between the comparison sample extracted feature and the training sample feature, using the obtained cosine similarity to characterize the difference between the training sample feature and the comparison sample extracted feature, so as to calculate the difference between the cosine similarity and the training label to obtain the feature extraction loss. When the comparison training sample is a positive comparison training sample, the training label is 1, and when the comparison training sample is a negative comparison training sample, the training label is 0.
[0137] In some embodiments, the computer device obtains the feature extraction loss based on the difference between the training sample feature and the comparison sample extracted feature, which can be specifically: the training sample feature and the comparison sample extracted feature are classified into two categories, similar and dissimilar, and the probability of similarity and dissimilarity are obtained, and the larger probability value is taken as the classification result, and the probability value of the classification result is used to characterize the difference between the training sample feature and the comparison sample extracted feature, and then the difference between the probability of the classification result and the training label can be calculated to obtain the feature extraction loss. When the comparison training sample is a positive comparison training sample, the training label is 1, and when the comparison training sample is a negative comparison training sample, the training label is 0.
[0138] In one embodiment, the computer device obtains the feature extraction loss based on the difference between the training sample features and the comparison sample extracted features by: calculating the feature distance between the training sample features and the comparison sample extracted features, using the feature distance to characterize the difference between the training sample features and the comparison sample extracted features, and using the feature distance as the feature extraction loss, wherein the feature distance can be, for example, the Euclidean distance or the L2 distance.
[0139] In the above embodiment, when the computer device trains the feature extraction model to be trained, it combines noise loss, feature extraction loss and classification loss to adjust parameters, so that the features extracted by the trained feature extraction model are more accurate.
[0140] In one embodiment, the comparison sample extraction features include positive sample extraction features corresponding to the positive comparison training samples and negative sample extraction features corresponding to the negative comparison training samples; obtaining the feature extraction loss based on the difference between the training sample features and the comparison sample extraction features includes: obtaining a positive feature difference value, which is a feature difference value between the training sample features and the positive sample extraction features; obtaining a negative feature difference value, which is a feature difference value between the training sample features and the negative sample extraction features; and determining the feature extraction loss based on the positive feature difference value and the negative feature difference value.
[0141] Specifically, the computer device extracts features from the positive comparison training samples based on the feature extraction model to be trained to obtain positive sample extraction features corresponding to the positive comparison training samples, extracts features from the negative comparison training samples based on the feature extraction model to be trained to obtain negative sample extraction features corresponding to the negative comparison training samples, obtains feature difference values between the training sample features and the positive sample extraction features to obtain positive feature difference values, obtains feature difference values between the training sample features and the negative sample extraction features to obtain negative feature difference values, and finally determines feature extraction loss based on the positive feature difference values and the negative feature difference values.
[0142] In one embodiment, the computer device may determine the feature extraction loss by referring to the following formula (1), where x a is the training sample feature, x p Extract features for positive samples, x n Extract features for negative samples, ||x a -x p || represents x a and x p The L2 distance between them, that is, the positive feature difference value, ||x a -x n || represents x a and x n The L2 distance between them, that is, the negative feature difference value, the purpose of formula (1) is to make the distance between the training sample and the negative comparison training sample larger than the distance between the training sample and the positive comparison training sample by α, where α is the margin (interval term), and the value of α can be set as needed.
[0143] L tri =max(||x a -x p ||-||x a -x n ||+α,0) Formula (1)
[0144] It can be seen from formula (1) that the feature extraction loss value is 0 only when the distance between the training sample and the negative comparison training sample is greater than the distance between the training sample and the positive comparison training sample by α, otherwise, the feature extraction loss value is greater than 0. Therefore, in the process of reducing the loss value, the distance between the training sample and the negative comparison training sample develops in the direction of being greater than the distance between the current training sample and the positive comparison training sample by α, so that the features extracted by the feature extraction model can better focus on the preservation of semantic features.
[0145] In one embodiment, negative comparison training samples can be obtained by the following steps: feature extraction of other training samples in the sample set to which the current training sample belongs, obtaining the extracted features of the comparison samples, calculating the feature distance between the training sample features corresponding to the current training sample and the extracted features of each comparison sample, and sorting the other training samples in the sample set from small to large according to the feature distance, so as to select training samples with relatively small feature distances as negative comparison training samples. The sample set to which the current training sample belongs can be pre-deduplicated, and by selecting training samples with relatively small feature distances as negative comparison training samples, difficult samples (Hard Sample) can be selected to alleviate the problem of sample imbalance, so that the model obtained by training is more accurate. Among them, difficult samples refer to these negative comparison training samples that belong to different categories from the current training sample but have small feature distances, which are difficult for classification. Among them, the deduplication operation refers to retaining one of the similar one or more training samples and removing the others.
[0146] In the above embodiment, since the feature extraction loss value is determined based on the positive feature difference value and the negative feature difference value, the feature extraction model takes into account the impact of too small inter-class feature distance on classification when performing similarity metric learning, thereby improving the semantic accuracy of the features extracted by the feature extraction model.
[0147] In one embodiment, the training sample features of the training samples are extracted by the feature extraction model to be trained, including: extracting the initial sample features of the training samples by the feature extraction model to be trained, and quantizing the initial sample features to obtain the training sample features of the training samples; training the feature extraction model according to the noise loss and the classification loss, and obtaining the target feature extraction model when the training stop condition is met, including: determining the quantization target corresponding to each quantization value in the training sample features based on a preset sign function, and determining the quantization loss based on the difference between each quantization value and the corresponding quantization target; statistically analyzing the quantization loss, the noise loss and the classification loss to obtain the target loss; adjusting the model parameters of the feature extraction model based on the target loss to obtain the adjusted feature extraction model; taking the adjusted feature extraction model as the feature extraction model to be trained, returning to extract the initial sample features of the training samples by the feature extraction model to be trained, and iteratively executing the steps of quantizing the initial sample features until the training stop condition is met to obtain the target feature extraction model.
[0148] Among them, quantization loss refers to the loss of calculating the quantization effect (whether it is close enough to -1 or 1). During the training process, it is expected that each quantization value in the training sample feature is close enough to 1 or -1. The quantization loss is positively correlated with the difference between the quantization value and the quantization target corresponding to the quantization value.
[0149] In one embodiment, for each training sample feature obtained by quantization, Q i is the value of the training sample feature Q at the i-th position obtained by quantizing the training sample, B i is the quantization target of the i-th position, B i By Q i After the preset sign function - sign function - is generated, the sign function can be used to generate each bit Q of the training sample feature Q by the following formula (3) i The target codes Bi are calculated respectively, and the final target code of Q is B. Then, the quantization loss is calculated by referring to formula (2).
[0150]
[0151]
[0152] In one embodiment, after obtaining the quantization loss, the computer device can perform a weighted summation of the quantization loss, noise loss, and classification loss by their respective weights to obtain the target loss, and then adjust the model parameters of the feature extraction model based on the target loss to obtain the adjusted feature extraction model.
[0153] In the above embodiment, the target loss is obtained by combining the quantization loss, the noise loss and the classification loss. The model is trained based on the target loss, and the feature extraction model obtained by training can extract accurate quantization features.
[0154] In a specific embodiment, Figure 6As shown, it is a schematic diagram of the training process of the feature extraction model. In this embodiment, the content of the training sample is an image, and the training sample is an image sample. The feature extraction model to be trained includes a CNN module and a hash quantization module. The computer device can input the training sample into the CNN module to obtain basic feature information, input the basic feature information into the hash quantization module, extract the embedding feature vector through the hash quantization module and perform quantization processing to obtain the training sample feature, and the training sample feature is further input into the multi-label classification model to obtain the classification result. The computer device can also input the candidate training samples corresponding to each candidate label in the candidate label set into the feature extraction model to be trained to obtain each candidate sample feature, and by inputting each candidate sample feature into the multi-prototype memory unit, multiple prototypes can be obtained. According to these prototypes and training sample features, recommended labels can be generated. In training, in addition to calculating the quantization loss, the original target loss (multi-label classification loss) is also used. The recommended label is used to perform adaptive adjustment with noise generalization to calculate the noise adaptive loss (noise-adapted loss), and the target loss is obtained based on the quantization loss and the noise adaptive loss. With the iteration of the model, the better the target multi-prototype representation effect, the higher the recognition accuracy rate is finally generated.
[0155] It should be noted that Figure 6 The multi-prototype memory unit in is the clustering unit mentioned above, which is used to cluster the features of each candidate sample to obtain a multi-prototype representation. Since the candidate sample features in this embodiment are extracted by the feature extraction model to be trained, the clustering unit can memorize the prototype representation of the label, learn the category prototype from the first iteration, and update the prototype in each iteration.
[0156] In one embodiment, Figure 7 As shown, a sample retrieval method is provided, and the method is described by taking the method executed by a computer device as an example. It can be understood that the computer device can be Figure 1 The terminal 102 shown may also be the server 104. In this embodiment, the sample retrieval method includes the following steps:
[0157] Step 702: Obtain query samples and candidate recall sample sets.
[0158] The candidate recall sample set refers to the set of recallable content samples in the database. The query sample refers to the content sample that needs to recall similar samples from the database. For example, given image A, similar images of image A are recalled from the database, and image A is the query sample.
[0159] Step 704: Input the query sample and the candidate recall sample in the candidate recall sample set into the target feature extraction model to obtain the query sample features corresponding to the query sample and the candidate recall sample features corresponding to the candidate recall sample.
[0160] Among them, the target feature extraction model is obtained by training the feature extraction model to be trained according to the noise loss and the classification loss; the classification loss value is determined based on the classification result and the category label of the training sample; the classification result is obtained by classifying the training sample based on the training sample feature; the training sample feature is obtained by extracting the feature of the training sample by the feature extraction model; the noise loss is determined based on the classification result and the recommended label when there is a difference between the recommended label and the category label; the recommended label is the candidate label to which the target representative sample feature determined from the candidate label set belongs; the target representative sample feature matches the training sample feature, and is selected from the representative sample features corresponding to each candidate label in the candidate label set; the candidate label set corresponds to the training sample.
[0161] Specifically, the computer device can input the query sample and each candidate recall sample in the candidate recall sample set into the target feature extraction model respectively, and perform feature extraction on the query sample and each candidate recall sample respectively through the target feature extraction model to obtain the query sample feature corresponding to the query sample and the candidate recall sample feature corresponding to each candidate recall sample.
[0162] Step 706: Determine a target retrieval sample corresponding to the query sample from the candidate recall sample set based on the query sample features and the candidate recall sample features.
[0163] In another embodiment, the computer device may calculate the similarity between the query sample feature and each candidate recall sample feature, and determine the candidate recall sample corresponding to the candidate recall sample feature whose similarity satisfies the similarity condition as the target retrieval sample. The similarity may be cosine similarity, and the similarity condition may be, for example, that the similarity is greater than a preset similarity threshold or that the similarity is ranked before a preset ranking threshold. For example, the calculated similarities are ranked from large to small to select the first N similarities, and the candidate recall samples corresponding to the candidate recall sample features corresponding to these similarities are determined as the target retrieval samples.
[0164] In another embodiment, the computer device may calculate the difference between the query sample feature and each candidate recall sample feature, and determine the candidate recall sample corresponding to the candidate recall sample feature whose difference satisfies the difference condition as the target retrieval sample. The difference may be a feature distance, and the difference condition may be, for example, that the difference is greater than a preset difference threshold or that the difference is ranked before a preset ranking threshold. For example, the calculated differences are ranked from small to large to select the first N differences, and the candidate recall samples corresponding to the candidate recall sample features corresponding to these differences are determined as the target retrieval samples.
[0165] In the above sample retrieval method, the target feature extraction model is obtained by training the feature extraction model to be trained based on the noise loss and the classification loss; the classification loss value is determined based on the classification result and the category label of the training sample; the classification result is obtained by classifying the training sample based on the training sample feature; the training sample feature is obtained by extracting the feature of the training sample through the feature extraction model; the noise loss is determined based on the classification result and the recommended label when there is a difference between the recommended label and the category label; the recommended label is the candidate label to which the target representative sample feature determined from the candidate label set belongs; the target representative sample feature matches the training sample feature, and is selected from the representative sample features corresponding to each candidate label in the candidate label set, and the candidate label set corresponds to the training sample. Therefore, the target feature extraction model is obtained by generalization learning based on the recommended label using a noise-friendly method, and the features extracted by the target feature extraction model have high accuracy, thereby improving the accuracy of sample retrieval.
[0166] In one embodiment, the sample retrieval method further includes: performing feature clustering on the candidate recall sample features of each candidate recall sample in the candidate recall sample set to obtain a plurality of cluster clusters; each cluster cluster has a corresponding cluster center; for each cluster center, establishing an association relationship between the cluster center and each candidate recall sample feature in the same cluster cluster; based on the query sample features and the candidate recall sample features, determining a target retrieval sample corresponding to the query sample from the candidate recall sample set, including: determining a target cluster center from each cluster center based on a feature distance between the query sample features and each cluster center; obtaining each candidate recall sample feature that has an association relationship with the target cluster center, and determining a target retrieval sample from each acquired candidate recall sample feature based on a feature distance between the query sample feature and each acquired candidate recall sample feature.
[0167] Specifically, after clustering, the computer device establishes the association between each cluster center and each candidate recall sample feature in the same cluster cluster, and then the cluster center can be used as the index of the cluster cluster, so that when searching, the feature distance between the query sample feature and these indexes can be calculated first, so as to screen out the cluster clusters where similar samples are located, and then search from the screened cluster clusters to obtain the target search sample. The index query method greatly reduces the amount of calculation in the search process and improves the search efficiency.
[0168] For example, assuming that the candidate recall sample set includes 1,000 candidate recall samples, and after clustering these candidate recall samples, 10 clusters are obtained. Then, when performing retrieval, the computer device can calculate feature clustering by comparing the query sample features with the cluster centers of these 10 clusters, select the top 3 clusters with the smallest feature distance, and calculate feature clustering by comparing the query sample features with the candidate recall sample features in these 3 clusters, thereby retrieving the target retrieval sample. No calculation is required for the remaining 7 clusters.
[0169] In one embodiment, the sample retrieval method further includes a feature extraction model processing step, which specifically includes: extracting training sample features of the training samples through the feature extraction model to be trained; the training samples have corresponding category labels; classifying the training samples based on the training sample features to obtain classification results, and determining the classification loss based on the classification results and the category labels; determining a set of candidate labels corresponding to the training samples, and obtaining representative sample features corresponding to each candidate label in the candidate label set; selecting representative sample features that match the training sample features from each representative sample feature, and determining the selected representative sample features as target representative sample features; determining the candidate labels to which the target representative sample features belong from the candidate label set, and determining the candidate labels to which the target representative sample features belong as recommended labels for the training samples; when there is a difference between the recommended label and the category label, determining the noise loss based on the classification results and the recommended label; training the feature extraction model based on the noise loss and the classification loss, and obtaining the target feature extraction model when the training stop condition is met.
[0170] The present application also provides an application scenario, which can be applied to the above-mentioned feature extraction model processing method and sample retrieval method. In this application scenario, the content is an image, the training sample is an image sample, and the feature extraction model processing method and the sample retrieval method can be applied to image retrieval. At present, the technology of image retrieval based on deep learning semantic hash feature image representation has gradually been applied in the industry. In large-scale semantic retrieval, the user gives a retrieval picture, such as a rock climbing picture, and the computer device extracts its hash features and compares them with the inventory samples, and finds the same rock climbing images and sorts them and returns them to the user. In addition to requiring large-scale triple samples to make the hash features have image measurement capabilities-the more similar the sample features, the higher the similarity, the more similar the sample features are, and a large number of semantic multi-labels need to be labeled. The hash features that are effective for large-scale inventory require a large number of images to be labeled, which is extremely expensive. An alternative solution is to label the inventory samples with the help of an existing or pre-trained multi-label classification model. The conventional method is to search for labels through an Internet search engine and use the images returned under the label as the training sample set of the label. This type of labeling method has a lot of noise: 1) label missing noise samples; 2) label error noise. Direct semantic hash learning under such samples can easily lead to deviations in image representation, resulting in representation failure, poor retrieval results and other problems. The difficulty lies in how to establish a more effective weakly supervised learning mechanism to improve the learning effect of the hash model under noisy samples.
[0171] To address this problem, this application proposes a feature extraction model processing method and a sample retrieval method, which can perform semantic modeling and identify sample noise based on memory units. Through dynamic noise recognition, multiple noise judgments are made on samples during the entire learning phase to avoid model collapse caused by a single noise recognition error. A regularization method is used to process noise samples to reduce the model's misjudgment of noise samples, thereby improving the effect of hash feature retrieval.
[0172] Specifically, the following description is made by taking the method executed by a computer device as an example. It can be understood that the computer device can be Figure 1 The terminal 102 shown may also be a server 104. The above feature extraction model processing method and sample retrieval method are applied in this application scenario as follows:
[0173] 1. Data Preparation
[0174] 1. Annotated data acquisition: Multi-label annotated image samples are obtained from Internet search engines. For the multi-label to be learned, the label is entered in the search engine, and the first 500 images returned are taken as images with the label. For example, if you search for golden retrievers in the search engine, save the first 500 images returned as image samples with the "golden retriever" label in the training set. Since there is no need to manually annotate each image, large-scale multi-label annotations are obtained only through search engines, which may cause: 1) The label is inconsistent with the searched image: For each searched image, due to the search engine's own errors or result sorting problems, it may not completely correspond to the searched label (for example, some of the first 500 images may not be golden retrievers), but most of the searched images are golden retrievers. 2) There may be unannotated relationships between images in all labels: an image searched for label A may also meet label B, but this image is not searched in label B, resulting in the image having both AB labels, but only annotated with label A. The above problems such as label errors and missing labels bring about the problem of weakly supervised learning.
[0175] 2. Training data mining: The training data includes training samples as anchors (anchor, hereinafter referred to as a), positive contrast training samples corresponding to the training samples (positive, hereinafter referred to as a), and negative contrast training samples corresponding to the training samples (negative, hereinafter referred to as n). a, p and n form a triplet (a, p, n). In the triplet, a and p constitute a positive sample pair, and a and n constitute a negative sample pair. In the learning task, the positive sample pair needs to have features that are close enough - the L2 distance is small enough (so that they can be retrieved from each other), while the negative sample pair needs to be far enough away. In this application scenario, images with the same semantics need to be able to be retrieved from each other.
[0176] Since the above step 1 generates image-label annotation information, the existing annotation information needs to be converted into triplets. The idea is: with the help of multi-label information, images with the same label can be positive sample pairs, and images with different labels can be negative sample pairs. Since the training requires that every bs image samples of the full data be input as a batch into the feature extraction model to be trained (hereinafter referred to as the model), the model uses the stochastic gradient descent (SGD) algorithm to learn each batch of samples until all images have been learned and an iteration is ended. In this application scenario, it is necessary to design a data extraction method for each batch to meet the needs of multi-label learning and hash feature metric learning. The specific steps include:
[0177] 2.1. Batch data composition: For a batch, randomly select 50 labels from all multi-labels, and randomly select 10 images from the images that have not been selected for each label, generating a total of 500 images, input into the model, and mark these 500 images as having been selected. When the number of selectable images under a certain label is less than 10, all selectable images of the label are taken (that is, the number of samples selected under a certain label can be less than 10).
[0178] 2.2. Mining triples from batch data: From the 50 labels corresponding to the above 500 images, perform the following operations:
[0179] a) For image a under a certain label A (as an anchor), randomly select one of the remaining 9 images under label A as the positive, randomly extract one label B from the remaining 49 labels, and randomly extract one image from the 10 images of B as the negative, thus generating a triplet. Repeat this step 5 times, so that each anchor can generate 5 triplets. (The number of repetitions can be adjusted, and the more images in the batch, the greater the number of repetitions).
[0180] B) All images under all labels are used as anchors in turn, and the above operation is performed to generate triplets. For all 500 images, a total of 2500 triplets are generated.
[0181] It can be understood that in the above process, since both positive and negative samples are randomly selected with replacement, the obtained triplet samples may be repeated (although the probability is very low). For the repeated triplet finally obtained, it can be directly removed and then participate in learning.
[0182] 2. Training Process
[0183] The model to be trained includes three parts: basic feature extraction module, hash quantization module, and classification model. The basic feature extraction model and hash quantization module together constitute the feature extraction model. The basic feature extraction model uses resnet101, and the parameters are shown in Table 1. It includes 6 parts, including convolution layer 1, convolution layer 2-convolution layer 5, and pooling layer. Convolution layer 1 is a 7×7×64 convolution with a stride of 2. Convolution layer 2 includes a 3×3 maximum pooling layer and 3 ResNet blocks. Convolution layer 3-convolution layer 5 include 3 ResNet modules, 4 ResNet modules, 23 ResNet modules, and 3 ResNet modules, respectively.
[0184] Table 1
[0185]
[0186]
[0187] The parameters of the hash quantization module are shown in Table 2, including a fully connected layer, which takes the output of the maximum pooling layer as input and outputs a 1x256 floating point. The floating point vector can be mapped to a binary vector (0 or 1) through the sign function, which is the hash feature in the final application. The parameters of the classification model are shown in Table 3, including a classification layer (Fc_class), the input of which is the output of the hash feature generation layer, and the output of which is the N-dimensional prediction probability. Among them, N is the number of classification categories.
[0188] Table 2
[0189] The name of the layer Output size layer Hash Quantization Layer 1x256 Fully connected layer
[0190] Table 3
[0191] The name of the layer Output size layer Classification layer 1xNclass Fully connected layer
[0192] It should be noted that the basic feature extraction model, hash quantization module and classification model can also use other model structures, such as the basic feature extraction model using resnet18CNN, and the hash quantization module using multi-layer fully connected layers. The specific training process is as follows:
[0193] 1. Parameter initialization:
[0194] In the pre-training phase, convolutional layers 1 to 5 use the parameters of ResNet101 pre-trained on the ImageNet dataset. Newly added layers such as feature layers are initialized with a Gaussian distribution with a variance of 0.01 and a mean of 0. The hash quantization layer and classification layer are initialized with a random normal distribution with a variance of 0.01 and a mean of 0.
[0195] 2. Set learning parameters: All parameters in Table 1, Table 2 and Table 3 need to be learned.
[0196] 3. Learning rate: The basic feature extraction model and hash quantization module use a learning rate of lr = 0.0005, and the classification model uses a learning rate of 0.005. After every 10 iterations, lr becomes 0.1 times the original value. When the gradient is fed back, the semantic loss (i.e., classification loss) first updates the parameters of the classification layer, and then feeds back to the hash quantization layer and the basic feature module; here, the learning rate of the quantized hash feature is set smaller than the semantic label (i.e., category label), which can avoid the multi-label semantic loss being fully transferred to the quantization layer, causing the semantic learning to over-affect the measurement effect of the quantization layer, thereby avoiding the semantic loss from over-affecting the measurement effect, and avoiding the dissimilar retrieval results due to semantic misjudgment.
[0197] 4. Learning process: Perform epoch iterations on the full data; process the full sample once in each iteration until the average epoch loss no longer decreases at a certain epoch, and obtain the image feature extraction model and the target multi-label classification model. The specific operations in each iteration of each epoch are as follows: Take the full image, generate each batch of images according to the above steps, and obtain the mined triples, and then:
[0198] 4.1. When epoch is greater than 1, initialize the memory unit. The initialization process includes: for the images of the full training set, from all labels, for labels with a data volume greater than 30, randomly select 200 images (60% of the images if the data volume is insufficient), and for labels with a data volume less than 30, randomly select 20 images to form the sample set Ms (Memory samples) for updating the memory unit. For Ms images, the hash features are obtained through the feature extraction model to be trained. For the images under each label in Ms, Kmeans clustering is performed, and the cluster center is 3, and the features of the 3 centers are obtained. The hash features of the 3 images closest to the 3 centers under the label are selected as the hash features of the 3 prototypes of the target label, which are the update results of the label in this memory unit. Here, the prototype of a label can be understood as the aggregation of different image contents under the label. For example, under the dog label, Border Collie, Husky, and Tibetan Mastiff are 3 prototypes. A label can have more than 3 prototypes, and the number of prototypes can be adjusted as needed.
[0199] 4.2. Forward calculation: Set all parameters in Table 1, Table 2 and Table 3 to the state that needs to be learned. During training, a forward calculation is performed on an input image to obtain the hash features and classification layer prediction results, which are represented by Q and P, where Q is a 1x256 vector representing the hash features, and P is a 1xNclass multi-label prediction, where each bit represents the predicted probability of the label corresponding to the position.
[0200] 4.3. Execute this step when epoch is greater than 1, otherwise skip it - Label prediction based on the target memory unit: perform L2 distance calculation on the hash features of all images in this bs and the hash features of all prototypes in the memory unit, and take the label of the prototype with the closest distance as the recommended label of the image. At the beginning of learning, due to the insufficient representation ability of hash features, the recommended labels may be inaccurate, but with the iterative optimization of the model, the recommended labels are getting better and better.
[0201] 4.4. Loss calculation: When the epoch is greater than 1, the recommended labels generated in the previous steps and the original labels (i.e., the label information obtained through the search engine) are used as joint supervision information, otherwise the original labels are used as supervision information. Specifically, for each batch, the total loss can be calculated by referring to the following formula (4):
[0202]
[0203] Among them, L triplet L is the loss corresponding to metric learning (i.e. finding similar and dissimilar parts in two images) (i.e. the feature extraction loss mentioned above). adapt-noise is the loss corresponding to semantic learning (i.e., finding commonalities of the same category in multiple images of the same category) (i.e., at least one of the classification loss and noise loss mentioned above), L coding is the loss corresponding to quantitative learning (i.e. quantizing features) (i.e. the quantization loss mentioned above). w1, w2 and w3 are weights, for example, w1 = 0.1, w2 = 1, w3 = 0.01 can be set.
[0204] The principle of weight setting here is: Since metric learning converges slowly and is much more difficult than semantic learning, semantic learning can accelerate convergence and avoid missing recall of semantically related images due to improper local representation in metric learning. However, fast semantic convergence can also easily lead to overfitting of quantitative features, which is constrained by setting the quantization learning rate to 0.1 times the semantics. Since the loss corresponding to quantitative learning converges faster than metric learning, and the importance of quantitative learning is lower than the feature measurement ability, in order to ensure that the loss of metric learning is dominant in the overall loss, so as to ensure that embedding always has the ability to measure similarity, w3 is set to 0.01 (or other values less than 1, which can be adjusted as needed).
[0205] The following is a detailed introduction to each loss:
[0206] 1) L triplet :Use the triples (a, p, n) mined from batch and refer to the formula (1) in the previous article for calculation. Among them, the distance between the samples a and n in the triples needs to be large enough to ensure that the triples can be distinguished in the quantization space, so the margin needs to be set larger. Considering that each bit of the 256-dimensional quantization vector will eventually be close to the value of -1 or 1, the preset reference margin = 160. ||x a -x p || represents the L2 distance between the quantized outputs Q of the two images a and p in the triplet. L triplet The purpose is to make the distance between a and n greater than the distance from p by margin.
[0207] 2) L coding: It is used to calculate the loss of quantization effect (whether it is close enough to -1 or 1) of the vector output by the hash quantization layer. Since the final application needs to be mapped to -1 / 1 binary value after quantization output, if each bit of Q output is close enough to 1 or -1, then the loss corresponding to the above metric learning is sufficient to describe the application of quantization features. Otherwise, even if the metric effect is good, it does not mean that the quantization representation in the application is good enough. Therefore, it is hoped that each bit of Q output is close enough to 1 or -1. For the quantization result of each image, refer to formula (2) and formula (3) in the previous article to calculate L coding Loss of Q i is the value of the quantized Q of the image at the i-th bit (a total of 256 bits in this case), B i is the quantization target of the i-th position, B i By Q i Generated by sign function - Use symbol function to generate target encoding for quantitative learning tasks.
[0208] It should be noted that the application directly uses the sign function to generate a quantized binary vector (the binary value is selected as 0 / 1, so that it is convenient to represent it with bits with values of 0 or 1 in the computer) for the output Q, which can be used for image retrieval.
[0209] 3) L adapt-noise :That is, the noise adaptive loss. For the probability vector output by the classification layer, the multi-label loss between it and the multi-label annotation (or joint multi-label recommendation) is calculated as follows. When epoch = 1, refer to the following formula (5) to calculate Lclass, that is, the multi-label classification loss; when epoch> 1, refer to the following formula (6) to calculate L adapt-noise For a sample i, where L bce0 is the BCE (binary cross entropy) loss with the original annotation as the target; L bce1 is the BCE loss with the recommended label as the target. For epochs greater than 1, the model has a certain representation ability and can recommend labels with relatively high accuracy. Since the noise of the recommended label and the original label cannot be absolutely determined, a learning method of generalization loss is designed in this embodiment. That is, multi-label loss is calculated for both the recommended label and the original label at the same time, and they are learned as two targets, so that the model does not only predict one label for the image, but can give two predicted labels.
[0210] L class =L bce0 (5)
[0211] L adapt-class =0.5*L bce0 +0.5*L bcel (6)
[0212] Ladapt-noise Another solution is to use multiple recommended labels, such as using two recommended labels and calculating the noise adaptive loss according to the following formula (7), where L bce2 is the recommended label of the second closest prototype in step 5.1 above (L bce1 is the recommended label for the first closest prototype). bce The calculation can be performed by referring to the following formula (8), where t[i] is the true value label vector and o[i] is the predicted probability output by the classification layer:
[0213] L adapt-class =0.3*L bce0 +0.3*L bce1 +0.3*L bce2 (7)
[0214]
[0215] It should be noted that for non-noise samples, the recommended multiple labels may all be original labels. In this case, the noise adaptation loss degenerates into the original multi-label loss, indicating that the original labels are relatively accurate. For noisy samples, the labels recommended multiple times may be different, or a sample may have both label 0 and label 1 and 2. Adopting noise adaptive learning is equivalent to learning 3 labels for the sample at the same time, and adding the sample with a weight of 0.3 as the importance to the whole batch image learning. The 0.3 sample weight is used to reduce the impact of the loss of the possible noise sample on the full amount of data.
[0216] Here, L bce1 , L bce2 Both are proxy tasks for noise processing, which can improve the generalization performance of the model for noise samples.
[0217] It is understandable that in other embodiments, the above-mentioned multi-label classification model can also be replaced by a multi-classification model. In this case, the multi-prototype representation obtained by the memory unit is obtained based on the multi-classification task. Then, when calculating the loss value, the multi-classification cross-entropy loss can be used instead of the above-mentioned BCE loss, and only one recommended label is needed in the classification.
[0218] It should be noted that the label in the multi-label in this application scenario refers to the category label, and multi-label classification is to classify the image into multiple categories. Multi-classification refers to determining one of the categories to which the image belongs from multiple classification categories.
[0219] 4.5. Model parameter update: Using the stochastic gradient descent algorithm, the target model loss value obtained in the previous step is subjected to gradient backward calculation to obtain the updated value of all model parameters, and the updated value is used to update the parameters of all models.
[0220] 3. Application of Quantitative Feature Retrieval
[0221] For all stock images, all Qs are extracted using the trained image feature extraction model, and the binary quantized features obtained by activation with the sign function are stored in the library. For the query image, Q is extracted using the above image feature extraction model and the binary quantized features are obtained by binary quantization, which are then compared one by one with the binary quantized features of the stock. The Hamming distance calculation for the binary quantized features can speed up the calculation efficiency (compared with the floating-point embedding features). After calculating the distance, the topK most similar ones are sorted from small to large and returned. In this way, stock images with semantic similarity to the query image can be retrieved (i.e., the target retrieval result samples mentioned above).
[0222] The present application also provides another application scenario, which is applicable to the above-mentioned feature extraction model processing method. In this application scenario, the image feature extraction model and the target multi-label classification model are trained through the same steps as the above application scenario, and the image feature extraction model and the target multi-label classification model are spliced to obtain the target classification model. According to the target classification model, multi-label classification of the input image can be realized.
[0223] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0224] Based on the same inventive concept, the embodiment of the present application also provides a feature extraction model processing device for implementing the feature extraction model processing method involved above, and a sample retrieval device for implementing the sample retrieval method involved above. The implementation solutions to the problems provided by the feature extraction model processing device and the sample retrieval device are similar to the implementation solutions recorded in the above method, so the specific limitations in one or more feature extraction model processing device embodiments and sample retrieval device embodiments provided below can refer to the limitations of the feature extraction model processing method and the sample retrieval method above, and will not be repeated here.
[0225] In one embodiment, Figure 8As shown, a feature extraction model processing device 800 is provided, comprising:
[0226] The sample feature extraction module 802 is used to extract the training sample features of the training sample through the feature extraction model to be trained; the training sample has a corresponding category label;
[0227] A classification loss calculation module 804 is used to classify the training samples based on the training sample features to obtain classification results, and determine the classification loss based on the classification results and the category labels;
[0228] The representative feature acquisition module 806 is used to determine the candidate label set corresponding to the training sample, and obtain the representative sample feature corresponding to each candidate label in the candidate label set;
[0229] A target feature determination module 808 is used to select a representative sample feature that matches the training sample feature from each representative sample feature, and determine the selected representative sample feature as a target representative sample feature;
[0230] The tag recommendation module 810 is used to determine the candidate tag to which the target representative sample feature belongs from the candidate tag set, and determine the candidate tag to which the target representative sample feature belongs as the recommended tag for the training sample;
[0231] A noise loss determination module 812, configured to determine a noise loss based on the classification result and the recommended label when there is a difference between the recommended label and the category label;
[0232] The training module 814 is used to train the feature extraction model according to the noise loss and the classification loss. When the training stop condition is met, a target feature extraction model is obtained; the target feature extraction model is used to extract sample features of the input sample.
[0233] The feature extraction model processing device extracts training sample features of training samples through the feature extraction model to be trained, wherein the training samples have corresponding category labels, classifies the training samples based on the training sample features, obtains classification results, determines the classification loss based on the classification results and the category labels, determines a set of candidate labels corresponding to the training samples, and obtains representative sample features corresponding to each candidate label in the candidate label set, selects representative sample features that match the training sample features from each representative sample feature, determines the selected representative sample features as target representative sample features, determines the candidate labels to which the target representative sample features belong from the candidate label set, and sets the target representative sample features as the target representative sample features. The candidate label to which the feature belongs is determined as the recommended label of the training sample. The recommended label is the possible correct label of the training sample. Therefore, when there is a difference between the recommended label and the category label, the training sample may be a noise sample. Since the noise situation of the recommended label and the original category label cannot be absolutely determined, the feature extraction model can be further trained according to the noise loss and classification loss. When the training stop condition is met, the target feature extraction model is obtained, so that the feature extraction model can use a noise-friendly method to perform generalization learning according to the recommended label, so that the processing of noise samples has a positive benefit to the model, thereby improving the accuracy of the features extracted by the feature extraction model.
[0234] In one embodiment, the representative feature acquisition module is also used to obtain the candidate sample sets corresponding to each candidate label; for each candidate label, feature extraction is performed on each candidate sample in the candidate sample set corresponding to the candidate label to obtain the features of each candidate sample corresponding to the candidate label; the features of each candidate sample corresponding to the candidate label are clustered, and the representative sample features corresponding to the candidate label are determined based on the clustering results.
[0235] In one embodiment, the representative feature acquisition module is also used to cluster each candidate sample feature to obtain multiple clusters; each cluster has a corresponding cluster center; for each cluster, from the candidate sample features corresponding to the cluster, select the candidate sample features whose feature distance with the cluster center meets the first distance condition; determine the selected candidate sample features as the representative sample features corresponding to the candidate label.
[0236] In one embodiment, the representative feature acquisition module is also used to input each candidate sample in the candidate sample set into the feature extraction model respectively; perform feature extraction on each candidate sample based on the feature extraction model to obtain the candidate sample features corresponding to each candidate sample; the training module is also used to perform the current round of training on the feature extraction model according to the noise loss and the classification loss to obtain the intermediate feature extraction model; use the intermediate feature extraction model as the feature extraction model to be trained, and enter the step of extracting the training sample features of the training samples through the feature extraction model to be trained to perform the next round of training, until the training stop condition is met, and the target feature extraction model is obtained.
[0237] In one embodiment, the target feature determination module is also used to obtain the feature distance between the training sample feature and each representative sample feature; from each representative sample feature, select the representative sample feature whose feature distance satisfies the second distance condition to obtain the target representative sample feature; the second distance condition includes at least one of the feature distance being less than a preset distance threshold or the feature distance being ranked before a preset ranking threshold.
[0238] In one embodiment, the target feature determination module is also used to sort the feature distances from small to large, so as to select the representative sample feature corresponding to the minimum feature distance to obtain the first target representative sample feature, and select the representative sample feature corresponding to the second minimum feature distance to obtain the second target representative sample feature; the label recommendation module is also used to determine the candidate label to which the first target representative sample feature belongs from the candidate label set to obtain the first recommended label, and determine the candidate label to which the second representative sample feature belongs to obtain the second recommended label.
[0239] In one embodiment, the noise loss determination module is also used to determine the first loss based on the difference between the first recommended label and the classification result, and determine the second loss based on the difference between the second recommended label and the classification result when there is a difference between the first recommended label and the category label and there is a difference between the second recommended label and the category label; and to calculate the first loss and the second loss to obtain the noise loss.
[0240] In one embodiment, the noise loss determination module is also used to compare the recommended label and the category label to obtain a comparison result; when the comparison result indicates that the recommended label and the category label are inconsistent, the noise loss is determined based on the classification result and the recommended label; the above-mentioned device also includes: a target loss calculation module, which is used to calculate the target loss based on the classification result and the category label when the comparison result indicates that the recommended label and the category label are consistent; the feature extraction model is trained based on the target loss, and when the training stop condition is met, the target feature extraction model is obtained.
[0241] In one embodiment, the training module is also used to obtain the comparison sample extraction features corresponding to the comparison training samples of the training samples, and obtain the feature extraction loss based on the difference between the training sample features and the comparison sample extraction features; obtain the target loss based on the noise loss, feature extraction loss and classification loss; adjust the parameters of the feature extraction model to be trained based on the target loss and continue training, and when the training stop condition is met, obtain the target feature extraction model.
[0242] In one embodiment, the comparison sample extraction features include positive sample extraction features corresponding to the positive comparison training samples and negative sample extraction features corresponding to the negative comparison training samples; the training module is also used to obtain positive feature difference values, which are feature difference values between the training sample features and the positive sample extraction features; obtain negative feature difference values, which are feature difference values between the training sample features and the negative sample extraction features; and determine feature extraction loss based on the positive feature difference values and the negative feature difference values.
[0243] In one embodiment, the sample feature extraction module is also used to extract the initial sample features of the training samples through the feature extraction model to be trained, and quantize the initial sample features to obtain the training sample features of the training samples; the training module is also used to determine the quantization target corresponding to each quantization value in the training sample features based on a preset sign function, and determine the quantization loss based on the difference between each quantization value and the corresponding quantization target; calculate the quantization loss, noise loss and classification loss to obtain the target loss; adjust the model parameters of the feature extraction model based on the target loss to obtain the adjusted feature extraction model; use the adjusted feature extraction model as the feature extraction model to be trained, return the initial sample features of the training samples extracted by the feature extraction model to be trained, and iterate the steps of quantizing the initial sample features until the training stop condition is met to obtain the target feature extraction model.
[0244] In one embodiment, Fig. 9 As shown, a sample retrieval device 900 is provided, comprising:
[0245] The sample acquisition module 902 is used to acquire a query sample and a candidate recall sample set;
[0246] The feature extraction module 904 is used to input the query sample and the candidate recall sample in the candidate recall sample set into the target feature extraction model to obtain the query sample feature corresponding to the query sample and the candidate recall sample feature corresponding to the candidate recall sample; wherein the target feature extraction model is obtained by training the feature extraction model to be trained according to the noise loss and the classification loss; the classification loss value is determined based on the classification result and the category label of the training sample; the classification result is obtained by classifying the training sample based on the training sample feature; the training sample feature is obtained by extracting the feature of the training sample by the feature extraction model; the noise loss is determined based on the classification result and the recommended label when there is a difference between the recommended label and the category label; the recommended label is the candidate label to which the target representative sample feature determined from the candidate label set belongs; the target representative sample feature matches the training sample feature, and is selected from the representative sample features corresponding to each candidate label in the candidate label set; the candidate label set corresponds to the training sample;
[0247] The retrieval module 906 is used to determine the target retrieval sample corresponding to the query sample from the candidate recall sample set based on the query sample features and the candidate recall sample features.
[0248] The above-mentioned sample retrieval device, since the target feature extraction model is obtained by training the feature extraction model to be trained based on the noise loss and the classification loss; the classification loss value is determined based on the classification result and the category label of the training sample; the classification result is obtained by classifying the training sample based on the training sample feature; the training sample feature is obtained by extracting the feature of the training sample through the feature extraction model; the noise loss is determined based on the classification result and the recommended label when there is a difference between the recommended label and the category label; the recommended label is the candidate label to which the target representative sample feature determined from the candidate label set belongs; the target representative sample feature matches the training sample feature, and is selected from the representative sample features corresponding to each candidate label in the candidate label set, and the candidate label set corresponds to the training sample, therefore, the target feature extraction model is obtained by generalization learning based on the recommended label using a noise-friendly method, and the accuracy of the features extracted by the target feature extraction model is high, thereby improving the accuracy of sample retrieval.
[0249] In one embodiment, the sample retrieval device further includes: an association relationship establishment module, which is used to perform feature clustering on the candidate recall sample features of each candidate recall sample in the candidate recall sample set to obtain multiple cluster clusters; each cluster cluster has a corresponding cluster center; for each cluster center, an association relationship is established between the cluster center and each candidate recall sample feature in the same cluster cluster; a retrieval module, which is used to determine a target cluster center from each cluster center based on the feature distance between the query sample feature and each cluster center; obtain each candidate recall sample feature that has an association relationship with the target cluster center, and determine a target retrieval sample from each acquired candidate recall sample feature based on the feature distance between the query sample feature and each acquired candidate recall sample feature.
[0250] Each module in the feature extraction model processing device and the sample retrieval device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0251] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store training samples. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a feature extraction model processing method or a sample retrieval method is implemented.
[0252] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.11As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a feature extraction model processing method or a sample retrieval method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.
[0253] Those skilled in the art will understand that Fig.10 and Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0254] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.
[0255] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0256] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.
[0257] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0258] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0259] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0260] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A feature extraction model processing method, characterized in that: The method comprises: Extracting training sample features of the training sample through the feature extraction model to be trained; the training sample has a corresponding category label; the category label is a label pre-labeled for the training sample; Inputting the training sample features into a classification model, obtaining an output of the classification model as a classification result, and determining a classification loss based on the classification result and the category label; Determine a candidate label set corresponding to the training sample, and obtain representative sample features corresponding to each candidate label in the candidate label set; the candidate label set corresponding to the training sample is a label set where the correct category label of the training sample is located; Selecting a representative sample feature that matches the training sample feature from each of the representative sample features, and determining the selected representative sample feature as a target representative sample feature; Determine the candidate label to which the target representative sample feature belongs from the candidate label set, and determine the candidate label to which the target representative sample feature belongs as the recommended label for the training sample; When the recommended label is consistent with the category label, determining the classification loss as the target loss; When there is a difference between the recommended label and the category label, determining a noise loss based on the difference between the classification result and the recommended label, and determining a target loss according to the noise loss and the classification loss; The feature extraction model is trained based on the target loss, and when a training stop condition is met, a target feature extraction model is obtained; the target feature extraction model is used to extract sample features of input samples.
2. The method according to claim 1, characterized in that The obtaining of representative sample features corresponding to each candidate tag in the candidate tag set includes: Obtain the candidate sample sets corresponding to each candidate label; For each candidate label, extract features of each candidate sample in the candidate sample set corresponding to the candidate label to obtain features of each candidate sample corresponding to the candidate label; Clustering is performed on each candidate sample feature corresponding to the candidate label, and based on the clustering result, a representative sample feature corresponding to the candidate label is determined.
3. The method according to claim 2, characterized in that The clustering of the candidate sample features corresponding to the candidate labels and determining the representative sample features corresponding to the candidate labels based on the clustering results includes: Clustering the features of each candidate sample to obtain multiple clusters; each cluster has a corresponding cluster center; For each cluster, selecting, from the candidate sample features corresponding to the cluster, a candidate sample feature whose feature distance with the cluster center satisfies a first distance condition; The selected candidate sample features are determined as representative sample features corresponding to the candidate labels.
4. The method according to claim 2, characterized in that: The extracting features of each candidate sample in the candidate sample set corresponding to the candidate label to obtain features of each candidate sample corresponding to the candidate label includes: Inputting each candidate sample in the candidate sample set into the feature extraction model respectively; Perform feature extraction on each of the candidate samples based on the feature extraction model to obtain candidate sample features corresponding to each of the candidate samples; The step of determining a target loss according to the noise loss and the classification loss, training the feature extraction model based on the target loss, and obtaining a target feature extraction model when a training stop condition is met includes: According to the noise loss and the classification loss, the feature extraction model is trained in a current round to obtain an intermediate feature extraction model; The intermediate feature extraction model is used as the feature extraction model to be trained, and the step of extracting training sample features of the training samples through the feature extraction model to be trained is entered to perform the next round of training, until the training stop condition is met, and the target feature extraction model is obtained.
5. The method according to claim 1, characterized in that: Selecting a representative sample feature that matches the training sample feature from each of the representative sample features, and determining the selected representative sample feature as a target representative sample feature, includes: Obtaining a feature distance between the training sample feature and each of the representative sample features; Selecting, from each of the representative sample features, a representative sample feature whose feature distance satisfies a second distance condition, to obtain a target representative sample feature; The second distance condition includes at least one of: the feature distance is less than a preset distance threshold; or the feature distance is ranked before a preset ranking threshold.
6. The method according to claim 5, characterized in that From each of the representative sample features, Selecting representative sample features whose feature distances satisfy the second distance condition to obtain target representative sample features includes: Sort the feature distances from small to large, select the representative sample feature corresponding to the minimum feature distance to obtain the first target representative sample feature, and select the representative sample feature corresponding to the second minimum feature distance to obtain the second target representative sample feature; The step of determining the candidate tag to which the target representative sample feature belongs from the candidate tag set, and determining the candidate tag to which the target representative sample feature belongs as the recommended tag for the training sample, includes: From the candidate tag set, determine the candidate tag to which the first target representative sample feature belongs to obtain a first recommended tag, and determine the candidate tag to which the second target representative sample feature belongs to obtain a second recommended tag.
7. The method according to claim 6, characterized in that When there is a difference between the recommended label and the category label, determining the noise loss based on the difference between the classification result and the recommended label comprises: When there is a difference between the first recommended tag and the category tag and there is a difference between the second recommended tag and the category tag, then determining a first loss based on a difference between the first recommended label and the classification result, and determining a second loss based on a difference between the second recommended label and the classification result; The first loss and the second loss are counted to obtain the noise loss.
8. The method according to claim 1, characterized in that The step of determining a target loss according to the noise loss and the classification loss, training the feature extraction model based on the target loss, and obtaining a target feature extraction model when a training stop condition is met includes: Obtaining a comparison sample extraction feature corresponding to a comparison training sample of the training sample, and obtaining a feature extraction loss based on a difference between the training sample feature and the comparison sample extraction feature; Obtaining a target loss based on the noise loss, the feature extraction loss, and the classification loss; The parameters of the feature extraction model to be trained are adjusted based on the target loss and the training is continued. When the training stop condition is met, the target feature extraction model is obtained.
9. The method according to claim 8, characterized in that The comparison sample extraction features include positive sample extraction features corresponding to positive comparison training samples and negative sample extraction features corresponding to negative comparison training samples; the feature extraction loss obtained based on the difference between the training sample features and the comparison sample extraction features includes: Obtaining a positive feature difference value, where the positive feature difference value is a feature difference value between the training sample feature and the positive sample extracted feature; Obtaining a negative feature difference value, wherein the negative feature difference value is a feature difference value between the training sample feature and the negative sample extracted feature; A feature extraction loss is determined based on the positive feature difference value and the negative feature difference value.
10. The method according to claim 1, characterized in that The step of extracting the training sample features of the training samples by using the feature extraction model to be trained includes: Extracting initial sample features of the training samples through the feature extraction model to be trained, and quantizing the initial sample features to obtain training sample features of the training samples; The step of determining a target loss according to the noise loss and the classification loss, training the feature extraction model based on the target loss, and obtaining a target feature extraction model when a training stop condition is met includes: Determine a quantization target corresponding to each quantization value in the training sample feature based on a preset sign function, and determine a quantization loss based on a difference between each quantization value and the corresponding quantization target; Counting the quantization loss, the noise loss, and the classification loss to obtain a target loss; Adjusting the model parameters of the feature extraction model based on the target loss to obtain an adjusted feature extraction model; The adjusted feature extraction model is used as the feature extraction model to be trained, and the initial sample features of the training samples extracted by the feature extraction model to be trained are returned, and the steps of quantizing the initial sample features are iteratively executed until the training stop condition is met to obtain the target feature extraction model.
11. A sample retrieval method, characterized in that: The method comprises: Obtain query samples and candidate recall sample sets; The query sample and the candidate recall sample in the candidate recall sample set are input into the target feature extraction model to obtain the query sample feature corresponding to the query sample and the candidate recall sample feature corresponding to the candidate recall sample; wherein the target feature extraction model is obtained based on target loss training, when the recommended label of the training sample is consistent with the category label, the target loss is the classification loss corresponding to the training sample, when there is a difference between the recommended label and the category label of the training sample, the target loss is determined based on the noise loss and classification loss corresponding to the training sample; the classification loss is determined based on the classification result and the category label of the training sample; the classification result is obtained by inputting the training sample feature into the classification model to obtain the input of the classification model. The training sample feature is obtained by extracting the training sample feature through the feature extraction model; the noise loss is determined based on the difference between the classification result and the recommended label; the recommended label is a candidate label to which the target representative sample feature determined from the candidate label set corresponding to the training sample belongs; the target representative sample feature matches the training sample feature and is selected from the representative sample features corresponding to each candidate label in the candidate label set; the candidate label set corresponds to the training sample; the category label of the training sample is a label pre-labeled for the training sample; the candidate label set corresponding to the training sample is a label set where the correct category label of the training sample belongs; Based on the query sample features and the candidate recall sample features, a target retrieval sample corresponding to the query sample is determined from the candidate recall sample set.
12. The method according to claim 11, characterized in that The method further comprises: Performing feature clustering on the candidate recall sample features of each candidate recall sample in the candidate recall sample set to obtain a plurality of clusters; each cluster has a corresponding cluster center; For each cluster center, establishing an association relationship between the cluster center and each candidate recall sample feature in the same cluster cluster; The determining, based on the query sample feature and the candidate recall sample feature, a target retrieval sample corresponding to the query sample from the candidate recall sample set comprises: Based on the feature distance between the query sample feature and each cluster center, determining a target cluster center from each cluster center; Each candidate recall sample feature associated with the target cluster center is obtained, and based on the feature distance between the query sample feature and each obtained candidate recall sample feature, a target retrieval sample is determined from each obtained candidate recall sample feature.
13. A feature extraction model processing device, characterized in that: The device comprises: A sample feature extraction module is used to extract training sample features of a training sample through a feature extraction model to be trained; the training sample has a corresponding category label; the category label is a label pre-marked for the training sample; A classification loss calculation module, used for inputting the training sample features into a classification model, obtaining the output of the classification model as a classification result, and determining the classification loss based on the classification result and the category label; A representative feature acquisition module is used to determine the candidate label set corresponding to the training sample and obtain the representative sample features corresponding to each candidate label in the candidate label set; the candidate label set corresponding to the training sample is the label set where the correct category label of the training sample is located; A target feature determination module is used to select a representative sample feature that matches the training sample feature from each of the representative sample features, and determine the selected representative sample feature as a target representative sample feature; A tag recommendation module, used to determine the candidate tag to which the target representative sample feature belongs from the candidate tag set, and determine the candidate tag to which the target representative sample feature belongs as the recommended tag for the training sample; a target loss determination module, configured to determine the classification loss as the target loss when the recommended label is consistent with the category label; and to determine the noise loss based on the difference between the classification result and the recommended label when there is a difference between the recommended label and the category label, and to determine the target loss based on the noise loss and the classification loss; A training module is used to train the feature extraction model based on the target loss, and obtain a target feature extraction model when a training stop condition is met; the target feature extraction model is used to extract sample features of input samples.
14. The device according to claim 13, characterized in that The representative feature acquisition module is also used to: obtain the candidate sample sets corresponding to each candidate tag; for each candidate tag, extract features of each candidate sample in the candidate sample set corresponding to the candidate tag to obtain the features of each candidate sample corresponding to the candidate tag; cluster the features of each candidate sample corresponding to the candidate tag, and determine the representative sample features corresponding to the candidate tag based on the clustering results.
15. The device according to claim 14, characterized in that The representative feature acquisition module is also used to: cluster each of the candidate sample features to obtain multiple clusters; each cluster has a corresponding cluster center; for each cluster, select a candidate sample feature whose feature distance with the cluster center meets a first distance condition from the candidate sample features corresponding to the cluster; determine the selected candidate sample feature as the representative sample feature corresponding to the candidate label.
16. The device according to claim 14, characterized in that The representative feature acquisition module is also used to: input each candidate sample in the candidate sample set into the feature extraction model respectively; extract features from each candidate sample based on the feature extraction model to obtain candidate sample features corresponding to each candidate sample; The training module is also used to perform a current round of training on the feature extraction model according to the noise loss and the classification loss to obtain an intermediate feature extraction model; use the intermediate feature extraction model as the feature extraction model to be trained, and enter the step of extracting training sample features of the training samples through the feature extraction model to be trained to perform the next round of training, until the training stop condition is met, and the target feature extraction model is obtained.
17. The device according to claim 13, characterized in that The target feature determination module is also used to: obtain the feature distance between the training sample feature and each of the representative sample features; select, from each of the representative sample features, a representative sample feature whose feature distance satisfies a second distance condition to obtain the target representative sample feature; the second distance condition includes at least one of the feature distance being less than a preset distance threshold or the feature distance being ranked before a preset ranking threshold.
18. The device according to claim 17, characterized in that The target feature determination module is also used to: sort the feature distances from small to large to select the representative sample feature corresponding to the minimum feature distance to obtain the first target representative sample feature, and select the representative sample feature corresponding to the second minimum feature distance to obtain the second target representative sample feature; the label recommendation module is also used to determine the candidate label to which the first target representative sample feature belongs from the candidate label set to obtain the first recommended label, and determine the candidate label to which the second target representative sample feature belongs to obtain the second recommended label.
19. The device according to claim 18, characterized in that The noise loss determination module is further used to: when there is a difference between the first recommended label and the category label and there is a difference between the second recommended label and the category label, determine a first loss based on the difference between the first recommended label and the classification result, and determine a second loss based on the difference between the second recommended label and the classification result; and calculate the first loss and the second loss to obtain the noise loss.
20. The device according to claim 13, characterized in that The training module is further used to: obtain a comparison sample extraction feature corresponding to a comparison training sample of the training sample, and obtain a feature extraction loss based on a difference between the training sample feature and the comparison sample extraction feature; A target loss is obtained based on the noise loss, the feature extraction loss and the classification loss; parameters of the feature extraction model to be trained are adjusted based on the target loss and training is continued, and when a training stop condition is met, a target feature extraction model is obtained.
21. The device according to claim 20, characterized in that The comparison sample extraction features include positive sample extraction features corresponding to positive comparison training samples and negative sample extraction features corresponding to negative comparison training samples; the training module is also used to: obtain positive feature difference values, which are feature difference values between the training sample features and the positive sample extraction features; obtain negative feature difference values, which are feature difference values between the training sample features and the negative sample extraction features; and determine feature extraction loss based on the positive feature difference values and the negative feature difference values.
22. The device according to claim 13, characterized in that The sample feature extraction module is further used to: extract initial sample features of the training sample through the feature extraction model to be trained, and quantify the initial sample features to obtain training sample features of the training sample; The training module is further used to: determine the quantization target corresponding to each quantization value in the training sample feature based on a preset sign function, and determine the quantization loss based on the difference between each quantization value and the corresponding quantization target; The quantization loss, the noise loss and the classification loss are counted to obtain a target loss; the model parameters of the feature extraction model are adjusted based on the target loss to obtain an adjusted feature extraction model; the adjusted feature extraction model is used as the feature extraction model to be trained, and the initial sample features of the training samples extracted by the feature extraction model to be trained are returned, and the steps of quantizing the initial sample features are iteratively executed until a training stop condition is met to obtain a target feature extraction model.
23. A sample retrieval device, characterized in that: The device comprises: A sample acquisition module is used to obtain query samples and candidate recall sample sets; A feature extraction module is used to input the query sample and the candidate recall sample in the candidate recall sample set into a target feature extraction model to obtain the query sample feature corresponding to the query sample and the candidate recall sample feature corresponding to the candidate recall sample; wherein the target feature extraction model is obtained based on target loss training, when the recommended label of the training sample is consistent with the category label, the target loss is the classification loss corresponding to the training sample, when there is a difference between the recommended label and the category label of the training sample, the target loss is determined based on the noise loss and classification loss corresponding to the training sample; the classification loss is determined based on the classification result and the category label of the training sample; the classification result is obtained by inputting the training sample feature into the classification model The output of the model; the training sample feature is obtained by extracting the features of the training sample by the feature extraction model; the noise loss is determined based on the difference between the classification result and the recommended label; the recommended label is a candidate label to which the target representative sample feature determined from the candidate label set corresponding to the training sample belongs; the target representative sample feature matches the training sample feature and is selected from the representative sample features corresponding to each candidate label in the candidate label set; the candidate label set corresponds to the training sample; the category label of the training sample is a label pre-annotated for the training sample; the candidate label set corresponding to the training sample is the label set where the correct category label of the training sample is located; The retrieval module is used to determine the target retrieval sample corresponding to the query sample from the candidate recall sample set based on the query sample characteristics and the candidate recall sample characteristics.
24. The device according to claim 23, characterized in that The device further includes: an association relationship establishment module, which is used to: perform feature clustering on the candidate recall sample features of each candidate recall sample in the candidate recall sample set to obtain multiple clusters; each cluster has a corresponding cluster center; for each cluster center, establish an association relationship between the cluster center and each candidate recall sample feature in the same cluster cluster; The retrieval module is also used to: determine a target cluster center from each cluster center based on the feature distance between the query sample feature and each cluster center; obtain each candidate recall sample feature that has an association relationship with the target cluster center, and determine a target retrieval sample from each candidate recall sample feature obtained based on the feature distance between the query sample feature and each acquired candidate recall sample feature.
25. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 or 11 to 12 are implemented.
26. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 or 11 to 12 are implemented.
27. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 or 11 to 12 are implemented.
Citation Information
Patent Citations
Underwater acoustic target radiation noise identification method based on domain adaptation
CN111709315A
Vehicle rear-shot image brand training method
CN113076840A
Product recommendation method, device and equipment based on an improved BERT model, and medium
CN113516533A
Data processing method and device and equipment
CN113705589A
Image processing method and device, computer equipment and readable storage medium
CN113705597A
