Feature extraction model processing, sample retrieval method, device and computer equipment

Abstract

The present application relates to a feature extraction model processing, sample retrieval method, device and computer equipment. The method includes: extracting features from training samples through a feature extraction model to be trained to obtain a training feature vector; performing classification prediction on the training samples based on the training feature vector to obtain a predicted semantic vector; obtaining a label semantic vector of a training category label corresponding to the training sample, the label semantic vector including at least two activated label vector components; determining an expected semantic vector of the training sample based on the label semantic vector, the expected semantic vector including an expected activation vector component corresponding to the position distribution of at least two activated label vector components; training the feature extraction model based on the difference between each expected activation vector component in the expected semantic vector and the predicted vector component at the corresponding position in the predicted semantic vector to obtain a target feature extraction model. The use of this method can improve the accuracy of feature extraction.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a feature extraction model processing method, apparatus, computer equipment, storage medium and computer program product, as well as a sample retrieval method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of computer technology, machine learning technology has emerged. Through machine learning, various machine learning models can be trained. For example, a feature extraction model for feature extraction can be trained. The feature extraction model can extract the features of the input sample, and the input sample can be identified based on the features to obtain the recognition result. For example, a sentence can be feature extracted to obtain a feature vector representing the sentence, and the sentence can be translated based on the feature vector.

[0003] In traditional technology, feature extraction models can be trained through training samples. However, it is often the case that the features extracted by the trained feature extraction models have relatively low accuracy. Summary of the invention

[0004] Based on this, it is necessary to provide a feature extraction model processing method, device, computer equipment, computer readable storage medium and computer program product that can improve the accuracy of feature extraction in response to the above technical problems.

[0005] On the one hand, the present application provides a feature extraction model processing method. The method includes: extracting features from training samples through a feature extraction model to be trained to obtain a training feature vector; performing classification prediction on the training samples based on the training feature vector to obtain a predicted semantic vector; obtaining a label semantic vector of a training category label corresponding to the training sample, the label semantic vector including at least two activated label vector components; determining an expected semantic vector of the training sample based on the label semantic vector, the expected semantic vector including an expected activation vector component corresponding to the position distribution of the at least two activated label vector components; determining a classification loss based on the difference between each of the expected activation vector components in the expected semantic vector and the predicted vector components at the corresponding positions in the predicted semantic vector; training the feature extraction model based on the classification loss, and obtaining a target feature extraction model when a training stop condition is met; the target feature extraction model is used to extract a sample feature vector of an input sample.

[0006] On the other hand, the present application also provides a feature extraction model processing device. The device includes: a feature extraction module, which is used to extract features from training samples through a feature extraction model to be trained to obtain a training feature vector; a classification prediction module, which is used to perform classification prediction on the training samples based on the training feature vector to obtain a predicted semantic vector; a label vector acquisition module, which is used to obtain a label semantic vector of a training category label corresponding to the training sample, wherein the label semantic vector includes at least two activated label vector components; an expected vector determination module, which is used to determine the expected semantic vector of the training sample based on the label semantic vector, wherein the expected semantic vector includes an expected activation vector component corresponding to the position distribution of the at least two activated label vector components; a classification loss determination module, which is used to determine the classification loss based on the difference between each of the expected activation vector components in the expected semantic vector and the predicted vector components at the corresponding positions in the predicted semantic vector; a model training module, which is used to train the feature extraction model based on the classification loss, and obtain a target feature extraction model when the training stop condition is met; the target feature extraction model is used to extract the sample feature vector of the input sample.

[0007] On the other hand, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the feature extraction model processing method when executing the computer program.

[0008] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the feature extraction model processing method are implemented.

[0009] On the other hand, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the feature extraction model processing method are implemented.

[0010] The feature extraction model processing method, device, computer equipment, storage medium and computer program product described above extract features from training samples through the feature extraction model to be trained to obtain training feature vectors, classify and predict the training samples based on the training feature vectors to obtain predicted semantic vectors, obtain label semantic vectors of training category labels corresponding to the training samples, determine expected semantic vectors of the training samples based on the label semantic vectors, and then determine the classification loss based on the differences between each expected activation vector component in the expected semantic vector and the predicted vector components at their corresponding positions in the predicted semantic vectors. The feature extraction model is trained through the classification loss, and the target feature extraction model finally obtained can learn features that tend to the label semantic vector, so that the semantic space where the label semantic vector is located can be simulated in the vector space where the training feature vector is located, so that the extracted feature vector can store feature information representing semantics, thereby improving the accuracy of feature extraction. Moreover, since the label semantic vector includes at least two activated label vector components, the training category labels can be more accurately represented, and the expected semantic vector is determined based on the label semantic vector and includes expected activation vector components corresponding to the position distribution of at least two activated label vector components, so that the calculated classification loss can be made more accurate, further improving the accuracy of feature extraction.

[0011] On the other hand, the present application provides a sample retrieval method. The method includes: obtaining a query sample and a candidate recall sample set; inputting the query sample and the candidate recall sample in the candidate recall sample set into a target feature extraction model respectively, and obtaining a query feature vector corresponding to the query sample and a candidate recall feature vector corresponding to the candidate recall sample; wherein the target feature extraction model is obtained by training the feature extraction model to be trained through classification loss, the classification loss is determined based on the difference between each of the expected activation vector components in the expected semantic vector and the predicted vector components at the corresponding positions in the predicted semantic vector, the predicted semantic vector is obtained by classifying and predicting the training sample based on the training feature vector, the expected semantic vector is determined based on the label semantic vector of the training category label corresponding to the training sample, the label semantic vector includes at least two activation label vector components, the expected semantic vector contains the expected activation vector component corresponding to the position distribution of the at least two activation label vector components, and the training feature vector is obtained by extracting features from the training sample by the feature extraction model to be trained; based on the query feature vector and the candidate recall feature vector, the target retrieval sample corresponding to the query sample is determined from the candidate recall sample set.

[0012] On the other hand, the present application also provides a sample retrieval device. The device comprises: a sample acquisition module, which is used to acquire a query sample and a candidate recall sample set; a feature extraction module, which is used to input the query sample and the candidate recall sample in the candidate recall sample set into a target feature extraction model respectively, to obtain a query feature vector corresponding to the query sample and a candidate recall feature vector corresponding to the candidate recall sample; wherein the target feature extraction model is obtained by training the feature extraction model to be trained through classification loss, the classification loss is determined based on the difference between each expected activation vector component in the expected semantic vector and the predicted vector component at the corresponding position in the predicted semantic vector, the predicted semantic vector is obtained by classifying and predicting the training sample based on the training feature vector, the expected semantic vector is determined based on the label semantic vector of the training category label corresponding to the training sample, the label semantic vector includes at least two activation label vector components, the expected semantic vector contains the expected activation vector component corresponding to the position distribution of the at least two activation label vector components, and the training feature vector is obtained by extracting features from the training sample by the feature extraction model to be trained; and a retrieval module, which is used to determine the target retrieval sample corresponding to the query sample from the candidate recall sample set based on the query feature vector and the candidate recall feature vector.

[0013] On the other hand, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above sample retrieval method are implemented.

[0014] On the other hand, the present application further provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above-mentioned sample retrieval method when executed by a processor.

[0015] On the other hand, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above sample retrieval method are implemented.

[0016] The above-mentioned sample retrieval method, apparatus, computer equipment, storage medium and computer program product, since the target feature extraction model is trained by the classification loss, the target feature extraction model finally obtained can learn the features that tend to the label semantic vector, so that the semantic space where the label semantic vector is located can be simulated in the vector space where the training feature vector is located, so that the extracted feature vector can store feature information representing the semantics, thereby improving the accuracy of feature extraction, and since the label semantic vector includes at least two activated label vector components, the training category label can be more accurately represented, and the expected semantic vector is determined based on the label semantic vector and includes the expected activation vector component corresponding to the position distribution of at least two activated label vector components, which can make the calculated classification loss more accurate, further improving the accuracy of feature extraction, and then using the feature vector extracted by the target feature extraction model for sample retrieval can improve the accuracy of sample retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 An application environment diagram of a feature extraction model processing method and a sample retrieval method in one embodiment;

[0018] Figure 2 A schematic diagram of a process flow of a feature extraction model processing method in one embodiment;

[0019] Figure 3 A mapping relationship diagram between a training feature vector and an expected semantic vector in one embodiment;

[0020] Figure 4 A comparison diagram of feature learning effects in one embodiment;

[0021] Figure 5 Schematic diagram of a training process of a feature extraction model in one embodiment;

[0022] Figure 6 is a flow chart of a sample retrieval method in one embodiment;

[0023] Figure 7 A schematic diagram of a specific process of searching by index in one embodiment;

[0024] Figure 8 is a structural block diagram of a feature extraction model processing device in one embodiment;

[0025] Fig. 9 is a structural block diagram of a sample retrieval device in one embodiment;

[0026] Fig.10 is an internal structure diagram of a computer device in one embodiment;

[0027] Fig.11FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0030] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0031] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further perform image processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, automatic driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0032] Machine Learning (ML) is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computer devices simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computer devices intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.

[0033] The solution provided in the embodiments of the present application involves artificial intelligence computer vision, machine learning and other technologies, which are specifically described by the following embodiments:

[0034] The feature extraction model processing method and sample retrieval method provided in this application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store data that the server 104 needs to process, such as training sample data. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 can be, but is not limited to, a laptop, a smart phone, a tablet computer, a desktop computer, a smart TV, a vehicle-mounted terminal, and a portable wearable device. An application can be provided on the terminal, through which the input sample can be retrieved to obtain the target retrieval sample. The application can refer to a client installed in the terminal, and the client (also referred to as an application client, APP client) refers to a program installed and running in the terminal; the application can also refer to an installation-free application, that is, an application that can be used without downloading and installing. This type of application is also commonly known as a mini-program, which usually runs in the client as a subroutine; the application can also refer to a web application opened through a browser; and so on. The server 104 can be implemented with an independent server or a server cluster or cloud server composed of multiple servers. It can be understood that the embodiment of the present application does not limit the number of terminals and servers. The terminal can be one or more, and the server can be one or more, which can be set as needed. Among them, multiple refers to at least two.

[0035] It is understandable that the feature extraction model processing method and sample retrieval method provided in the embodiment of the present application can be executed by the terminal 102, can be executed by the server 104, or can be executed collaboratively by the terminal 102 and the server 104. For example, the server 104 can obtain the target feature extraction model through the feature extraction model processing method provided in the embodiment of the present application, and send the target feature extraction model to the terminal 102, and the terminal 102 can implement sample retrieval based on the target feature extraction model.

[0036] In one embodiment, Figure 2 As shown, a feature extraction model processing method is provided, which is executed by a computer device, and the computer device can be Figure 1 The terminal 102 in Figure 1 The server 104 in the embodiment may also be a system composed of the terminal 102 and the server 104. Specifically, the feature extraction model processing method includes the following steps:

[0037] Step 202: extract features from the training samples using the feature extraction model to be trained to obtain a training feature vector.

[0038] Among them, the training sample refers to the content sample used to train the feature extraction model, and the content can be any one of text, audio or image. There is a corresponding training category label for the training sample, and the training category label corresponding to the training sample is used to identify the category of the training sample. For example, if the training sample is an image, the training category label corresponding to the training sample can be animal species such as dogs, cats, fish, plant species such as corals, pine trees, osmanthus, or object types such as magnifying glasses, cabinets, and water bottles. The category label corresponding to the training sample can be one or more, and multiple means at least two. When there are multiple category labels for the training sample, it means that the training sample can be classified into multiple different categories. Taking the training sample as a picture as an example, if the training sample contains both cats and dogs, then the training sample can be classified into the cat category or the dog category.

[0039] The feature extraction model to be trained refers to the feature extraction model that needs parameter adjustment. The feature extraction model is a machine learning model used to extract features and output feature vectors. The feature extraction model includes at least an embedding model, which is used to output feature vectors. The feature vector output by the embedding model can be called an embedding vector, that is, an embedding representation vector. The embedding model can be a model composed of one or more fully connected layers.

[0040] In one embodiment, the embedding model can normalize the output feature vector so that each feature component of the feature vector has a value range of -1 to 1. In other embodiments, the embedding model can also perform binary quantization on the feature vector to obtain binary quantized features. Binary quantization refers to the process of binary encoding the feature. For example, the feature can be encoded as a binary code with values ​​of 0 and 1. Bit compression can also be performed during the encoding process. For example, the feature vector can be compressed to 48 bits. The feature obtained by binary quantization of the feature vector can be called a hash feature. In this case, the embedding model can be called a hash quantization model.

[0041] In one embodiment, the feature extraction model may only include an embedding model, and the input end of the embedding model may be connected to the output end of the trained basic neural network model, and receive the output of the basic neural network model as input. The basic neural network model may be a model for extracting feature information contained in the content, and the basic neural network model may be a neural network based on artificial intelligence, such as a convolutional neural network (CNN), or a network such as ResNet101 (deep residual network 101) or ResNet18 (deep residual network 18). The convolutional neural network is a type of feedforward neural network (Feedforward Neural Networks) that includes convolution calculations and has a deep structure. Specifically, after obtaining the training sample, the computer device inputs the training sample into the trained basic neural network model, extracts feature information through the trained basic neural network model, and inputs the extracted feature information into the embedding model to be trained to obtain a training feature vector. For example, taking the training sample as an image, the image can be input into the trained CNN model to extract image features, and the extracted image features can be input into the embedding model to obtain the training feature vector of the image.

[0042] In other embodiments, the feature extraction model may include a basic neural network model and an embedding model, in which case the basic neural network model and the embedding model are trained together as feature extraction models. Specifically, after acquiring the training sample, the computer device inputs the training sample into the basic neural network model to be trained, extracts feature information through the basic neural network model to be trained, and inputs the extracted feature information into the embedding model to be trained to obtain a training feature vector.

[0043] It can be understood that different feature extraction models can be trained for different types of samples. For example, if the sample is an image type sample, then a feature extraction model for extracting image features can be trained. If the sample is a speech type sample, then a feature extraction model for extracting speech features can be trained.

[0044] In one embodiment, the computer device locally stores a set of labeled training samples. The computer device can obtain training samples from the locally stored training sample set, input the training samples into the feature extraction model to be trained, and output the training feature vector corresponding to the training sample through the feature extraction model. It can be understood that for the same training sample, different training feature vectors can be obtained under different feature extraction models. For example, the training feature vector can be a D-dimensional feature vector directly output by the embedding model, or a feature obtained by normalizing the feature vector, or a hash feature.

[0045] In one embodiment, the training samples in the training sample set can be labeled by searching from an Internet search engine. For example, taking the training samples as images, specifically, for a certain label, the label can be entered in the search engine, and the first N images returned are taken as images with the label. For example, if a search engine searches for golden retrievers, the first 500 images returned are saved as image samples with the label "golden retriever" in the training sample set.

[0046] Step 204: classify and predict the training samples based on the training feature vector to obtain a predicted semantic vector.

[0047] Among them, classification prediction refers to predicting the category to which the training sample belongs based on the training feature vector to determine the specific category to which the training sample belongs. The predicted semantic vector refers to the probability vector output by the classification prediction, which can represent the category information of the training sample and is a probability vector containing semantic information.

[0048] Specifically, the computer device can perform classification prediction on the training samples based on the training feature vector to obtain a predicted semantic vector.

[0049] In one embodiment, the computer device can use a classification model to classify the training feature vector, that is, input the training feature vector into the classification model, and output a predicted semantic vector through the classification model. Among them, the classification model refers to a machine learning model that can perform category recognition. In some embodiments, the classification model may include one or more fully connected layers. When the classification model includes multiple fully connected layers, the first few fully connected layers can extract semantic features from the training feature vector, so it can be called a semantic layer Semantic_layer, and the last fully connected layer is used for classification, which is called a classification layer. The number of semantic layers can be determined as needed, and the semantic layer can be deepened for situations where more feature intersections are required. By setting the semantic layer, high-order features can be abstracted to more fully mine the classification information.

[0050] Step 206: Obtain a label semantic vector of a training category label corresponding to the training sample, where the label semantic vector includes at least two activated label vector components.

[0051] Among them, the label semantic vector is used to characterize the semantics of the training category label. The label vector component refers to the vector component contained in the label semantic vector. For example, assuming that the label semantic vector is (1, 1, 0, 0), then 1, 1, 0, and 0 are label vector components. The activated label vector component refers to the activated label vector component in the label semantic vector, that is, the label vector component is the activation value. For example, 1 can be used to represent activation and 0 can be used to represent inactivation, then the label semantic vector is (1, 1, 0, 0) including two activated label vector components. In an embodiment of the present application, for each training category label, a label semantic vector including at least two activated label vector components is used to express its semantic information, which can express semantic information more specifically and accurately than one-hot encoding (one-bot).

[0052] Specifically, the training samples belong to a training sample set, the training category labels of each training sample in the training sample set constitute a label set, label semantic vectors are generated for each training category label in the label set in advance, and an association relationship between the label semantic vector and the training category label is established. Therefore, when training is performed, the computer device can obtain the label semantic vector of the training category label from the association relationship between the pre-established training category label and the label semantic vector.

[0053] In one embodiment, each training category label in the label set can be split into elements, each training category label can be split into multiple elements, and the co-occurring elements between each training category label can be extracted, such as forests and parks have grass elements, and the vector dimension of the label semantic vector is determined based on the number of all extracted elements. Each element corresponds to a feature bit, and the feature bit refers to the sorting position of the feature component, so that the label semantic vector of each training category label can be generated. For example, assuming that the label set includes three training category labels, where training category label 1 includes three elements (a, b, c), training category label 2 includes three elements (a, d, e), and training category label 3 includes three elements (c, e), then it can be determined that the label semantic vector of training category label 1 is (1, 1, 1, 0, 0), the label semantic vector of training category label 2 is (1, 0, 0, 1, 1), and the label semantic vector of training category label 3 is (0, 0, 1, 0, 1).

[0054] In another embodiment, the computer device can also generate an initial semantic vector for each training list tag in the tag set through the trained word2vec, and compare each feature component in the initial semantic vector with the semantic activation threshold. If a feature component is greater than the activation threshold, the feature component is determined as an activated feature component, and the value of the feature bit is set to an activation value, which may be 1, for example. If a feature component is less than the activation threshold, the feature component is determined as an inactivated feature component, and the value of the feature bit is set to an inactivated value, which may be 0, to finally obtain a tag semantic vector.

[0055] Step 208: determining an expected semantic vector of the training sample based on the label semantic vector, wherein the expected semantic vector includes an expected activation vector component corresponding to the position distribution of at least two activation label vector components.

[0056] Among them, the expected semantic vector refers to the semantic vector expected to be output when the training sample is classified and predicted. The semantic vector can be used as supervisory information in the training process so that the feature extraction model can learn semantic information in the training process. The expected activation vector component refers to the activated expected vector component. The expected vector component here refers to the vector component contained in the expected semantic vector. For example, assuming that the expected semantic vector is (1, 0, 0, 1), then 1, 1, 0, 0 are the expected vector components. The expected semantic vector contains the expected activation vector component corresponding to the position distribution of at least two activated label vector components, that is, if there is an activated label vector component in a certain feature position in the label semantic vector, then there is an expected activation vector component at the corresponding position in the expected semantic vector. For example, assuming that the label semantic vector is (1, 1, 0, 0), the label semantic vector has an activation vector component 1 in both the first feature position and the second feature position, then in the expected semantic vector, the first feature position and the second feature position must be 1.

[0057] Specifically, the training category label corresponding to the training sample is used to characterize the category to which the training sample belongs. The label semantic vector of the training category label can express the semantic information of the training sample. Therefore, based on the label semantic vector, the computer device can determine the expected semantic vector of the training sample. Since the expected semantic vector contains expected activation vector components corresponding to the position distribution of at least two activated label vector components, it can contain the semantic information expressed by the label semantic vector.

[0058] In one embodiment, when a training sample corresponds to a training category label, the semantic information represented by the training category label is the semantic information contained in the training sample, so the computer device can use the label semantic vector as the expected semantic vector of the training sample.

[0059] In another embodiment, when the training sample corresponds to multiple training category labels, the expected semantic vector needs to be able to express the semantic information of each training category label. At this time, the computer device can fuse the semantic vectors of each label to obtain the expected semantic vector that can jointly express the semantic information of each training category label.

[0060] In one embodiment, when the computer device fuses the semantic vectors of each tag, it ensures that the dimension of the expected semantic vector is consistent with the dimension of the tag semantic vector to avoid the dimension of the expected semantic vector being too large due to an excessive number of tags.

[0061] Step 210, determining the classification loss based on the difference between each expected activation vector component in the expected semantic vector and the predicted vector component at the corresponding position in the predicted semantic vector.

[0062] The position correspondence between the expected activation vector component and the predicted vector component means that the sorting position of the expected activation vector component in the expected semantic vector is the same as the sorting position of the predicted vector component in the predicted semantic vector. For example, assuming that the expected semantic vector is (a1, a2, a3, a4) and the predicted semantic vector is (b1, b2, b3, b4), then a1 corresponds to b1, a2 corresponds to b2, a3 corresponds to b3, and a4 corresponds to b4.

[0063] Specifically, since the expected semantic vector contains expected activation vector components corresponding to the position distribution of at least two activation label vector components, the expected semantic vector contains at least two expected activation vector components. For each expected activation vector component, the computer device can calculate the difference between the expected activation vector component and the predicted vector component at the corresponding position in the predicted semantic vector, and perform statistics on the differences corresponding to each expected activation vector component to determine the classification loss.

[0064] It is understandable that the expected semantic vector may also include unexpected activation vector components. For each unexpected activation vector component, the computer device can calculate the difference between the unexpected activation vector component and the predicted vector component at the corresponding position in the predicted semantic vector, and finally count the differences corresponding to each expected activation vector component and the differences corresponding to each unexpected activation vector component to obtain the classification loss.

[0065] Step 212, training the feature extraction model based on the classification loss, and obtaining a target feature extraction model when the training stop condition is met; the target feature extraction model is used to extract the sample feature vector of the input sample.

[0066] Specifically, the computer device can adjust the model parameters of the feature extraction model based on the classification loss and continue training until the training stop condition is met, thereby obtaining a target feature extraction model. The obtained target feature extraction model is used to extract a sample feature vector of the input sample, through which similar samples can be retrieved from a sample database, and through which the input sample can also be classified and identified to determine the category to which the input sample belongs.

[0067] In one embodiment, during the training process, the computer device may use any one of the stochastic gradient descent algorithm, Adagrad (Adaptive Gradient) algorithm, Adadelta (an improvement of the AdaGrad algorithm), RMSprop (an improvement of the AdaGrad algorithm), Adam (Adaptive Moment Estimation) algorithm, etc. to adjust the model parameters of the feature extraction model.

[0068] In one embodiment, the training stop condition may be that the model parameters no longer change, the loss reaches the minimum value, or the number of training times reaches the maximum number of iterations, etc. In other embodiments, the computer device may perform multiple rounds (epochs) of iterative training on the feature extraction model, and after each round of training, the target loss of that round is obtained. The training stop condition may be that the average target loss in a certain epoch no longer decreases.

[0069] In the above-mentioned feature extraction model processing method, the above-mentioned feature extraction model processing method, device, computer equipment, storage medium and computer program product, extract features from training samples through the feature extraction model to be trained to obtain training feature vectors, classify and predict training samples based on the training feature vectors to obtain predicted semantic vectors, obtain label semantic vectors of training category labels corresponding to the training samples, determine expected semantic vectors of the training samples based on the label semantic vectors, and then determine the classification loss based on the differences between each expected activation vector component in the expected semantic vector and the predicted vector components at their corresponding positions in the predicted semantic vector, train the feature extraction model through the classification loss, and finally The obtained target feature extraction model can learn features that tend to the label semantic vector, so that the semantic space where the label semantic vector is located can be simulated in the vector space where the training feature vector is located, so that the extracted feature vector can store feature information representing the semantics, thereby improving the accuracy of feature extraction. Moreover, since the label semantic vector includes at least two activated label vector components, the training category labels can be more accurately represented. The expected semantic vector is determined based on the label semantic vector and includes expected activation vector components corresponding to the position distribution of at least two activated label vector components, which can make the calculated classification loss more accurate, further improving the accuracy of feature extraction.

[0070] In one embodiment, the training sample corresponds to multiple training category labels, and based on the label semantic vector, determining the expected semantic vector of the training sample includes: forming a feature component set from label vector components with the same sorting position in each label semantic vector; for a feature component set that includes an activated label vector component, setting the expected vector component at the sorting position to which the feature component set belongs to an activated value; for a feature component set that does not include an activated label vector component, setting the expected vector component at the sorting position to which the feature component set belongs to an inactive value; and combining the expected vector components according to their corresponding sorting positions to form the expected semantic vector of the training sample.

[0071] The sorting position refers to the position of the label vector component in the label semantic vector. For example, for the label semantic vector (a, b, c), the sorting position of a is the first, the sorting position of b is the second, and the sorting position of c is the third.

[0072] Specifically, the computer device can group the label vector components with the same sorting position in each label semantic vector into a feature component set, so that corresponding to each sorting position, a feature component set containing multiple label vector components can be obtained. For each feature component set, the computer device can determine whether the feature component set contains an activated label vector component. If the feature component set contains an activated label vector component, the expected vector component at the sorting position to which the feature component set belongs can be set to an activated value. If the feature component set does not contain an activated label vector component, the expected vector component at the sorting position to which the feature component set belongs can be set to an inactive value. Finally, the expected vector components are combined according to their corresponding sorting positions to form the expected semantic vector of the training sample.

[0073] For example, assume that the training samples correspond to two training category labels A and B, where the label semantic vector of A is (1, 0, 1, 0), and the label semantic vector of B is (0, 0, 1, 1). The feature component set with the first sorting position includes the activated label vector component 1, then the expected vector component at this sorting position is 1, the feature component set with the second sorting position does not include the activated label vector component, then the expected vector component at this sorting position is 0, the feature component set with the third sorting position includes the activated label vector component 1, then the expected vector component at this sorting position is 1, the feature component set with the fourth sorting position includes the activated label vector component 1, then the expected vector component at this sorting position is 1, and the final expected semantic vector is (1, 0, 1, 1).

[0074] In the above embodiment, by setting the expected vector component at the sorting position of the feature component set that includes the activated label vector component to an activated value, and setting the expected vector component at the sorting position of the feature component set that does not include the activated label vector component to an inactive value, the obtained expected semantic vector can represent infinite multiple labels through limited feature bits, thereby saving the amount of calculation in the loss calculation process and improving training efficiency.

[0075] In one embodiment, the classification loss is determined based on the difference between each expected activation vector component in the expected semantic vector and the predicted vector components at the corresponding positions in the predicted semantic vector, including: for each expected vector component in the expected semantic vector, based on the difference between the expected vector component and the predicted vector component corresponding to the sorted position in the predicted semantic vector, determining the initial value of the classification loss component at the sorted position where the expected vector component is located; counting the activated label vector components in the feature component set at the sorted position where the expected vector component is located, and determining the activation degree of the expected vector component at the sorted position based on the statistical results; weighting the initial value of the classification loss component based on the activation degree to obtain a target value of the classification loss component; and counting the target values ​​of each classification loss component to determine the classification loss.

[0076] The statistics may be to count the number of activated label vector components. The activation degree obtained based on the statistical result is positively correlated with the statistical result, that is, the larger the statistical value, the higher the activation degree, and vice versa, the smaller the statistical value, the lower the activation degree.

[0077] Specifically, after the computer device determines the initial value of the classification loss component at the sorting position where each expected vector component is located, for each sorting position, it can count the number of activated label vector components in the feature component set at the sorting position to obtain a quantity value, and then determine the degree of activation at the sorting position based on the quantity value, use the degree of activation as the weight of the initial value of the classification loss component at the sorting position, multiply the weight by the initial value of the classification loss component to obtain a target value of the classification loss component, and finally the computer device can add the target values ​​of each classification loss component to determine the classification loss.

[0078] In one embodiment, a computer device determines the activation level of a sample feature component at a sorting position based on statistical results. For a feature component set that includes an activated label vector component, the statistically obtained quantity value is used as the activation level of the feature component set at the sorting position; and for a feature component set that does not include an activated label vector component, the quantity value is 0, and the activation level of the feature component set at the sorting position can be determined to be 1.

[0079] In other embodiments, after obtaining the quantitative values ​​through statistics, the computer device may further normalize the quantitative values ​​to obtain the final activation degree, wherein, in order to avoid ignoring the loss component at the sorting position with a quantitative value of 0, the quantitative values ​​obtained through statistics at each sorting position may be added with a preset bias value to obtain a target quantitative value, and the activation degree may be (target quantitative value / sum of target quantitative values ​​corresponding to each sorting position), wherein the preset bias value may be 1, for example. For example, in the above example, for training category labels A and B, the quantitative values ​​corresponding to each sorting position may be 1, 0, 2, and 1, and after adding the preset bias value 1, the target quantitative values ​​are 2, 1, 3, and 2, respectively, and the activation degrees calculated at the end are 1 / 4, 1 / 8, 3 / 8, and 1 / 4, respectively.

[0080] In one embodiment, the computer device may perform a cross entropy loss calculation based on the difference between the expected vector component and the predicted vector component corresponding to the sorted position in the predicted semantic vector to obtain an initial value of the classification loss component at the sorted position where the expected vector component is located.

[0081] In the above embodiment, by counting the activated label vector components in the feature component set at the sorting position where the expected vector component is located, the activation degree of the sorting position where the expected vector component is located is determined based on the statistical results, and the initial value of the classification loss component is weighted based on the activation degree to obtain the target value of the classification loss component. The obtained loss classification target value can accurately reflect the sharing of the classification loss by the initial values ​​of each classification loss component, and then the target values ​​of each classification loss component are counted to determine the classification loss, so that a more accurate classification loss can be obtained.

[0082] In one embodiment, before training the feature extraction model based on the classification loss and obtaining the target feature extraction model when the training stop condition is met, the above method also includes: determining the semantic constraint loss based on the difference between the training feature vector of the training sample and the expected semantic vector, and the semantic constraint loss is used to semantically constrain the training feature vector extracted by the feature extraction model; training the feature extraction model based on the classification loss and obtaining the target feature extraction model when the training stop condition is met, including: determining the target loss based on the semantic constraint loss and the classification loss; training the feature extraction model based on the target loss and obtaining the target feature extraction model when the training stop condition is met.

[0083] Among them, the semantic constraint loss is positively correlated with the difference between the training feature vector of the training sample and the expected semantic vector, and is used to semantically constrain the training feature vector extracted by the feature extraction model. By semantically constraining the training feature vector, the training feature vector can learn the distribution of the semantic information in the expected semantic vector, making the semantic information expressed by the training feature vector more accurate.

[0084] Specifically, the computer device can calculate the regression loss based on the difference between the training feature vector and the expected semantic vector of the training sample to obtain the semantic constraint loss, and then determine the target loss based on the semantic constraint loss and the classification loss. Finally, the parameters of the feature extraction model are adjusted based on the target loss and the loss is continued until the training stop condition is reached, and the target feature extraction model is obtained.

[0085] In the above embodiment, by calculating the semantic constraint loss and combining the semantic constraint loss with the classification loss for model training, the feature extraction model can learn more accurate semantic information, thereby further improving the accuracy of feature extraction.

[0086] It is understandable that in order to ensure that the extracted training feature vector can accurately learn the semantic information distribution in the expected semantic vector, it is necessary to ensure that the dimension of the training feature vector is greater than or equal to the dimension of the expected semantic vector. Based on this, the present application provides the following embodiments.

[0087] In one embodiment, the semantic constraint loss is determined based on the difference between the training feature vector and the expected semantic vector of the training sample, including: if the dimension of the training feature vector is consistent with the dimension of the expected semantic vector, then the vector distance between the training feature vector and the expected semantic vector is calculated, and the vector distance is used as the semantic constraint loss corresponding to the training sample.

[0088] In this embodiment, if the dimension of the training feature vector is consistent with the dimension of the expected semantic vector, then for each training vector component in the training feature vector, there is a corresponding expected vector component in the expected semantic vector, that is, there is a one-to-one mapping relationship between the training vector components in the training feature vector and the expected vector components in the expected semantic vector. Figure 3 In Figure (a), it is assumed that the dimensions of the training feature vector and the expected semantic vector are both 3. The black circles in Figure (a) represent the expected vector components in the expected semantic vector, and the white circles represent the training vector components in the training feature vector. It can be seen from Figure (a) that there is a one-to-one mapping relationship between the training vector components and the expected vector components.

[0089] In this embodiment, considering that the training feature vector and the expected semantic vector are vectors of the same dimension, the computer device can directly calculate the vector distance between the training feature vector and the expected semantic vector to obtain the regression loss, thereby obtaining the semantic constraint loss corresponding to the training sample. In a specific implementation, the vector distance can be, for example, the L2 distance.

[0090] In the above embodiment, the semantic constraint loss is obtained by training the vector distance between the feature vector and the expected semantic vector, which can improve the calculation efficiency of the regression loss and thus improve the training efficiency.

[0091] Considering that when characterizing samples, it is necessary to characterize the samples as a whole, and the classification information is only a highly abstract sample. For different samples under the same category, it is necessary to introduce sample-specific information after the semantic information representing the classification to distinguish different samples under the same category. Therefore, the dimension of the training feature vector can be greater than the dimension of the expected semantic vector to reserve certain feature bits as a supplement to the classification information. Based on this, the present application further provides the following embodiments.

[0092] In one embodiment, a semantic constraint loss is determined based on the difference between a training feature vector and an expected semantic vector of a training sample, including: if the dimension of the training feature vector is greater than the dimension of the expected semantic vector, then for each expected vector component in the expected semantic vector, a training vector component is selected from the training feature vector, and the selected training vector component is used as a mapping vector component of the expected vector component; the difference between the expected vector component and the mapping vector component is calculated to obtain a semantic constraint loss component corresponding to the expected vector component; and each semantic constraint loss component is counted to obtain a semantic constraint loss.

[0093] In this embodiment, the dimension of the training feature vector is greater than the dimension of the expected semantic vector. Then, when calculating the semantic constraint loss, for each expected vector component in the expected semantic vector, the training vector component corresponding to the expected vector component can be first determined from the training feature vector, and the determined training vector component is used as the mapping vector component of the expected vector component. That is, in the learning process, the feature bits of each mapping vector component are used to learn the corresponding expected vector components. Then, the regression loss calculation can be performed based on the difference between the expected vector component and the mapping vector component to obtain the semantic constraint loss component corresponding to the expected vector component. Finally, the various semantic constraint loss components are counted to obtain the semantic constraint loss component. The statistics here can be summed or averaged.

[0094] It can be understood that the feature bits where the unselected training vector components in the training feature vector are located are representation bits that can be used to supplement the sample information other than the category information.

[0095] In one embodiment, for each expected vector component in the expected semantic vector, the computer device can determine the training vector component corresponding to the expected vector component from the training feature vector according to a certain rule. For example, for each expected vector component, the training vector component corresponding to its position can be selected from the training feature vector, and the selected training vector component is used as the mapping vector component of the expected vector component. For example, assuming that the training feature vector is a 5-dimensional vector and the expected semantic vector is a 3-dimensional vector, for the expected vector component on the first feature position in the expected semantic vector, the training vector component on the first feature position in the training feature vector is used as the mapping vector component, for the expected vector component on the second feature position in the expected semantic vector, the training vector component on the second feature position in the training feature vector is used as the mapping vector component, and for the expected vector component on the third feature position in the expected semantic vector, the training vector component on the third feature position in the training feature vector is used as the mapping vector component.

[0096] It is understandable that in some other embodiments, the computer device may also select training vector components according to other rules, as long as it is ensured that each expected vector component has a corresponding mapping vector component.

[0097] In the above embodiment, if the dimension of the training feature vector is greater than the dimension of the expected semantic vector, the difference between the expected vector component and the mapping vector component is calculated to obtain the semantic constraint loss component corresponding to the expected vector component, and the various semantic constraint loss components are counted to obtain the semantic constraint loss. The semantic constraint loss can be accurately calculated to improve the accuracy of feature extraction.

[0098] In one embodiment, a training vector component is selected from the training feature vector, and the selected training vector component is used as the mapping vector component of the expected vector component, including: grouping the training vector components in the training feature vector according to the dimension of the expected semantic vector to obtain a plurality of vector component groups whose number is consistent with the dimension of the expected semantic vector; for each expected vector component in the expected semantic vector, selecting a vector component group from the plurality of feature component groups as a target vector component group; and selecting a training vector component from the target vector component group as the mapping vector component corresponding to the expected vector component.

[0099] Specifically, the computer device can group the training vector components in the training feature vector according to the dimension of the expected semantic vector to obtain multiple vector component groups whose number is consistent with the dimension of the expected semantic vector. The vector component groups can be sorted according to the sorting positions of the vectors they contain, so that for each expected vector component in the expected semantic vector, the computer device can select the vector component group corresponding to its sorting position from the multiple feature component groups as the target vector component group, and select a training vector component from the target vector component group as the mapping vector component corresponding to the expected vector component.

[0100] In one embodiment, if the dimension Nh of the training feature vector is an integer multiple of the dimension n of the expected semantic vector, that is, n*P=Nh (P is greater than 1), the training vector components in the training feature vector can be grouped evenly. Figure 3 Figure (b) in the figure, where the expected semantic vector is a 3-dimensional vector and the training feature vector is 6-dimensional. The black circles in (b) represent the expected vector components in the expected semantic vector, and the white circles represent the training vector components in the training feature vector. In (b), the training vector components in the training feature vector are divided into three groups in sequence, with every two training vector components forming a group. The first training vector component in each group is used as the mapping vector component corresponding to the expected vector component.

[0101] In the above embodiment, by grouping the training vector components in the training feature vector and selecting a training vector component from each group as the mapping vector component corresponding to the expected vector component, each feature bit in the training feature vector can better learn the semantic information in the expected semantic vector.

[0102] It can be seen from the above embodiments that the vector dimension of the training feature vector is greater than or equal to the dimension of the expected semantic vector, and the expected semantic vector is obtained based on the training category label. When there are multiple training category labels in the training sample, the expected semantic vector needs to be obtained by fusing multiple label semantic vectors. Therefore, the dimension of the expected semantic vector is greater than or equal to the label semantic vector. Based on this, the present application also provides the following embodiments to generate label semantic vectors.

[0103] In one embodiment, the training samples belong to a training sample set, and the training category labels corresponding to the training samples in the training sample set constitute a label set. The above method also includes: constructing a target semantic space; the target vector dimension of the target semantic space does not exceed the vector dimension of the training feature vector; and mapping each training category label in the label set to the target semantic space to obtain a label semantic vector for each training category label.

[0104] Specifically, the computer device can construct a target semantic space and ensure that the target vector dimension of the target semantic space does not exceed the vector dimension of the training feature vector, and then map each training category label in the label set to the target semantic space to obtain the label semantic vector of each training category label. The computer device can further establish a mapping relationship between the training category label and the label semantic vector, so that during the training process, the computer device can obtain the label semantic vector of the training category label based on the mapping relationship.

[0105] Furthermore, in a specific embodiment, constructing a target semantic space includes: generating a target matrix whose matrix order matches the target vector dimension, the target matrix including multiple candidate representation vectors, and the multiple candidate representation vectors form a target semantic space; the candidate representation vector includes a first numerical value and a second numerical value of equal quantity; mapping each training category label in the label set to the target semantic space to obtain a label semantic vector for each training category label, including: for each semantic category label in the semantic category label set, selecting a candidate representation vector from multiple candidate representation vectors as the label semantic vector corresponding to each semantic category label.

[0106] The target matrix includes a plurality of candidate representation vectors, and the candidate representation vectors include a first value and a second value of equal quantity, the first value and the second value are different values, the first value can be used as an activation vector component, and the second value can be used as an inactivation vector component. Since the candidate representation vector includes a first value and a second value of equal quantity, when the candidate representation vector is used as a label semantic vector, the constructed semantic space is a uniformly distributed semantic space.

[0107] Specifically, the computer device may determine the dimension of the target vector of the target semantic space, and after determining the dimension of the target vector, generate a target matrix whose matrix order matches the dimension of the target vector, and the target matrix may be a Hadamard matrix. It is known that for an n-order Hadamard matrix H, its n+1-order Hadamard matrix is:

[0108]

[0109] Therefore, we can get Hadamard matrices of various orders, for example, the 4th order Hadamard matrix is ​​as follows:

[0110]

[0111] Each value in the matrix can be 1 or -1, 1 means activation, -1 means inactivation. At the same time, the first row is all 1, and the matrix is ​​a symmetric matrix, and the sum of the elements on the diagonal of the matrix (the trace of the matrix) is 0.

[0112] Except for the first row, the vectors in other rows of the Hadamard matrix can be used as candidate representation vectors. These candidate representation vectors constitute a target semantic space. The computer device can select a candidate representation vector from the target semantic space as a label semantic vector corresponding to the semantic category label. For different semantic category labels, the computer device selects different candidate label vectors as label semantic vectors.

[0113] Since the number of -1 and 1 in each candidate representation vector of the Hadamard matrix is ​​the same, when these candidate representation vectors are used as vectors in the target semantic space, the effect of uniform activation can be achieved, and the target semantic vector formed is a uniformly distributed semantic space. In an embodiment of the present application, by mapping the training category label to the uniformly activated semantic space, the obtained semantic label vector can make the feature vector extracted by the trained feature extraction model simulate the uniformly distributed semantic space, improve the semantic representation effect of the features extracted by the feature extraction model, and avoid redundancy or compression of certain semantic information in the feature position.

[0114] refer to Figure 4 , is a feature learning effect comparison diagram in one embodiment, where the left diagram represents a uniformly distributed semantic space, different circles represent the distribution of different labels in the semantic space, the largest circle is the overall semantic space, and different labels are distributed on the circumference. Since the semantic space uses a vector in the Hadamard matrix to represent each label, the distance between any two labels is the same. The right diagram represents a hash space. In this embodiment, the hash space is used for metric learning, so similar samples are close to each other in the hash space, for example Figure 4 The two underlined Xs in the figure represent similar sample pairs. Figure 4 In (a) of the figure, the uniformly distributed semantic space is not directly related to the hash similarity space, so samples of different categories are mixed and distributed in the hash space. As shown in (a), the solid X represents the sample belonging to the label represented by the small circle 1, and the dotted X represents the sample belonging to the label represented by the small circle 2. It can be seen from (a) that these samples are randomly distributed, and the samples represented by the dotted X and the solid X intersect with each other. Figure 4 As shown in Figure (b), in the hash space based on semantic space fine-tuning, in addition to the sample pairs being close to each other, each sample pair is distributed in the direction of the label semantic vector where its label is located, so a space close to semantic uniform distribution can be simulated in the hash space.

[0115] In the above embodiment, by simulating a uniformly distributed semantic space, the semantic representation effect of the extracted features can be further improved.

[0116] In one embodiment, a feature extraction model is trained based on classification loss, and when a training stop condition is met, a target feature extraction model is obtained, including: obtaining a contrast feature vector corresponding to a contrast training sample of the training sample, and obtaining a feature extraction loss based on the difference between the training feature vector and the contrast feature vector; obtaining a target loss based on the feature extraction loss and the classification loss; adjusting parameters of the feature extraction model based on the target loss and continuing training, and obtaining a target feature extraction model when the training stop condition is met.

[0117] The comparison training sample refers to a training sample used for comparison with the training sample to determine the feature extraction loss. The comparison training sample may include at least one of a positive comparison training sample or a negative comparison training sample, wherein the positive comparison training sample refers to a training sample similar to the training sample, and the negative comparison training sample refers to a training sample that is not similar to the current training sample.

[0118] Specifically, the computer device can perform feature extraction on the comparison training samples based on the feature extraction model to be trained, obtain the training feature vector corresponding to the comparison training sample as the comparison feature vector, obtain the feature extraction loss based on the difference between the training feature vector and the comparison feature vector, perform weighted summation of the feature extraction loss and the classification loss to obtain the target loss, adjust the parameters of the feature extraction model to be trained based on the target loss and continue training, and obtain the target feature extraction model when the training stop condition is met.

[0119] In one embodiment, the computer device obtains the feature extraction loss based on the difference between the training feature vector and the comparison feature vector by: calculating the cosine similarity between the comparison feature vector and the training feature vector, using the obtained cosine similarity to characterize the difference between the two, and thus calculating the difference between the cosine similarity and the training label to obtain the feature extraction loss. When the comparison training sample is a positive comparison training sample, the training label is 1, and when the comparison training sample is a negative comparison training sample, the training label is 0.

[0120] In some embodiments, the computer device obtains the feature extraction loss based on the difference between the training feature vector and the comparison feature vector by: classifying the training feature vector and the comparison feature vector by similarity, dividing them into two categories, similar and dissimilar, obtaining the probability of similarity and the probability of dissimilarity, taking the larger probability value as the classification result, and using the probability value of the classification result to represent the difference between the training feature vector and the comparison feature vector, and then calculating the difference between the probability of the classification result and the training label to obtain the feature extraction loss. Wherein, when the comparison training sample is a positive comparison training sample, the training label is 1, and when the comparison training sample is a negative comparison training sample, the training label is 0.

[0121] In one embodiment, the computer device obtains the feature extraction loss based on the difference between the training feature vector and the comparison feature vector by: calculating the feature distance between the training feature vector and the comparison feature vector, using the feature distance to characterize the difference between the training feature vector and the comparison feature vector, and using the feature distance as the feature extraction loss, wherein the feature distance can be, for example, the Euclidean distance or the L2 distance.

[0122] In the above embodiment, when the computer device trains the feature extraction model to be trained, it adjusts the parameters of the feature extraction loss and the classification loss at the same time, so that the features extracted by the trained feature extraction model are more accurate.

[0123] In one embodiment, the comparison feature vector includes a positive comparison feature vector corresponding to a positive comparison training sample and a negative comparison feature vector corresponding to a negative comparison training sample; obtaining a feature extraction loss based on the difference between the training feature vector and the comparison feature vector includes: obtaining a positive feature difference value, which is a feature difference value between the training feature vector and the positive comparison feature vector; obtaining a negative feature difference value, which is a feature difference value between the training feature vector and the negative comparison feature vector; and determining a feature extraction loss based on the positive feature difference value and the negative feature difference value.

[0124] Specifically, the computer device extracts features from the positive contrast training samples based on the feature extraction model to be trained to obtain the positive contrast feature vector corresponding to the positive contrast training samples, extracts features from the negative contrast training samples based on the feature extraction model to be trained to obtain the negative contrast feature vector corresponding to the negative contrast training samples, obtains the feature difference value between the training feature vector and the positive contrast feature vector to obtain the positive feature difference value, obtains the feature difference value between the training feature vector and the negative contrast feature vector to obtain the negative feature difference value, and finally determines the feature extraction loss based on the positive feature difference value and the negative feature difference value.

[0125] In one embodiment, the computer device may determine the feature extraction loss by referring to the following formula (1), where x a is the training feature vector, x p is the positive contrast feature vector, x n is the negative contrast eigenvector, ||x a -x p || represents x a and x p The L2 distance between them, that is, the positive feature difference value, ||x a -x n || represents x a and x nThe L2 distance between them, that is, the negative feature difference value, the purpose of formula (1) is to make the distance between the training sample and the negative comparison training sample larger than the distance between the training sample and the positive comparison training sample by α, where α is the margin (interval term), and the value of α can be set as needed.

[0126] L tri =max(||x a -x p ||-||x a -x n ||+α,0) Formula (1)

[0127] It can be seen from formula (1) that the feature extraction loss value is 0 only when the distance between the training sample and the negative comparison training sample is greater than the distance between the training sample and the positive comparison training sample by α, otherwise, the feature extraction loss value is greater than 0. Therefore, in the process of reducing the loss value, the distance between the training sample and the negative comparison training sample develops in the direction of being greater than the distance between the training sample and the positive comparison training sample by α, so that the features extracted by the feature extraction model can express the semantic information more accurately.

[0128] In the above embodiment, since the feature extraction loss value is determined based on the positive feature difference value and the negative feature difference value, the feature extraction model takes into account the impact of the small inter-class feature distance on classification when performing similarity metric learning, thereby further improving the semantic accuracy of the features extracted by the feature extraction model.

[0129] It can be understood that when the feature extraction loss calculated in combination with the above formula (1) is used to train the feature extraction model, the feature extraction model can perform metric learning, so that the extracted features have similarity measurement capabilities.

[0130] In one embodiment, a training feature vector of a training sample is extracted through a feature extraction model to be trained, including: extracting initial sample features of the training sample through the feature extraction model to be trained, and quantizing the initial sample features to obtain a training feature vector of the training sample; training the feature extraction model based on the classification loss, and when the training stop condition is met, obtaining a target feature extraction model, including: determining a quantization target corresponding to each quantization value in the training feature vector based on a preset sign function, and determining a quantization loss based on the difference between each quantization value and the corresponding quantization target; obtaining a target loss based on the quantization loss and the classification loss; adjusting parameters of the feature extraction model based on the target loss and continuing training, and obtaining a target feature extraction model when the training stop condition is met.

[0131] Among them, quantization loss refers to the loss of calculating the quantization effect (whether it is close enough to -1 or 1). During the training process, it is expected that each quantization value in the training feature vector is close enough to 1 or -1. The quantization loss is positively correlated with the difference between the quantization value and the quantization target corresponding to the quantization value.

[0132] In one embodiment, for each training sample, the training feature vector obtained by quantization processing, Q i is the value of the training feature vector Q at the i-th position obtained by quantizing the training sample, B i is the quantization target of the i-th position, B i By Q i After the preset sign function - sign function - is generated, the sign function can be used to train each bit of the feature vector Q by the following formula (3): i Calculate the target code Bi respectively, and finally the target code of Q is B, and then refer to formula (2) to calculate the quantization loss.

[0133]

[0134] In one embodiment, after obtaining the quantization loss, the computer device can perform a weighted summation of the quantization loss and the classification loss by using their respective preset weights to obtain the target loss, and then adjust the model parameters of the feature extraction model based on the target loss to obtain the adjusted feature extraction model.

[0135] In the above embodiment, the target loss is obtained by combining the quantization loss and the classification loss, and the model is trained based on the target loss. The feature extraction model obtained by training can extract accurate quantization features.

[0136] In a specific embodiment, Figure 5 , which is a schematic diagram of the training process of the feature extraction model. In this embodiment, the content of the training sample is an image, and the training sample is an image sample. The feature extraction model to be trained includes a CNN module, an embedding module, and a hash quantization module. Figure 5The computer device first establishes a uniformly expressed semantic multi-label space corresponding to the training sample set with the help of the Hadamard matrix. During the training process, the training sample is input into the CNN module to obtain basic feature information, and the basic feature information is input into the embedding module to obtain an embedding vector, and then the training feature vector is quantized by the hash quantization module. The training feature vector is further input into the classification model, and the classification loss is calculated based on the predicted semantic vector output by the classification model and the expected semantic vector of the training sample, and the semantic constraint loss is calculated based on the difference between the expected semantic vector and the training feature vector. The semantic constraint loss is weighted with the metric loss and the classification loss to obtain the target loss. Among them, the metric loss can be obtained by referring to the weighted sum of the quantization loss and the feature extraction loss in the above embodiment.

[0137] In this embodiment, in addition to learning the metric capability in the hash feature learning space, each sample is used to simulate a semantic space with uniform semantic distribution, so that the hash feature of each image is adjusted toward the uniformly distributed label semantic vector while satisfying the metric learning similarity. Compared with the hash and semantic dual space model, the multi-bit activation effect of the hash feature is more effectively improved, thereby alleviating the problem of over-concentration on certain feature information or excessive redundancy of hash bits. Figure 5 The hash code metric constraint loss is an important condition for fine-tuning the output of the hash layer in the non-semantic space by using a model that uniformly distributes the semantic space, thereby supporting the learning of simulating the semantic space in the non-semantic space and fine-tuning the hash representation of the non-semantic space.

[0138] In one embodiment, Figure 6 As shown, a sample retrieval method is provided, and the method is described by taking the method executed by a computer device as an example. It can be understood that the computer device can be Figure 1 The terminal 102 shown may also be a server 104, or a system composed of the terminal 102 and the server 104. In this embodiment, the sample retrieval method includes the following steps:

[0139] Step 602: Obtain query samples and candidate recall sample sets.

[0140] The candidate recall sample set refers to the set of recallable content samples in the database. The query sample refers to the content sample that needs to recall similar samples from the database. For example, given image A, similar images of image A are recalled from the database, and image A is the query sample.

[0141] Step 604 , input the query sample and the candidate recall samples in the candidate recall sample set into the target feature extraction model respectively, to obtain the query feature vector corresponding to the query sample and the candidate recall feature vector corresponding to the candidate recall sample.

[0142] Among them, the target feature extraction model is obtained by training the feature extraction model to be trained through classification loss, the classification loss is determined based on the difference between each expected activation vector component in the expected semantic vector and the predicted vector component at the corresponding position in the predicted semantic vector, the predicted semantic vector is obtained by classifying and predicting the training sample based on the training feature vector, the expected semantic vector is determined based on the label semantic vector of the training category label corresponding to the training sample, the label semantic vector includes at least two activated label vector components, the expected semantic vector contains expected activation vector components corresponding to the position distribution of at least two activated label vector components, and the training feature vector is obtained by extracting features from the training sample by the feature extraction model to be trained.

[0143] Specifically, the computer device can input the query sample and each candidate recall sample in the candidate recall sample set into the target feature extraction model respectively, and perform feature extraction on the query sample and each candidate recall sample respectively through the target feature extraction model to obtain the query feature vector corresponding to the query sample and the candidate recall feature vector corresponding to each candidate recall sample.

[0144] Step 606: Based on the query feature vector and the candidate recall feature vector, determine the target retrieval sample corresponding to the query sample from the candidate recall sample set.

[0145] In one embodiment, the computer device may calculate the similarity between the query feature vector and each candidate recall feature vector, and determine the candidate recall sample corresponding to the candidate recall feature vector whose similarity satisfies the similarity condition as the target retrieval sample. The similarity may be cosine similarity, and the similarity condition may be, for example, that the similarity is greater than a preset similarity threshold or that the similarity is ranked before a preset ranking threshold. For example, the calculated similarities are ranked from large to small to select the first N similarities, and the candidate recall samples corresponding to the candidate recall feature vectors corresponding to these similarities are determined as the target retrieval samples.

[0146] In another embodiment, the computer device may calculate the difference between the query feature vector and each candidate recall feature vector, and determine the candidate recall sample corresponding to the candidate recall feature vector whose difference satisfies the difference condition as the target retrieval sample. The difference may be a feature distance, and the difference condition may be, for example, that the difference is less than a preset difference threshold or that the difference is ranked before a preset ranking threshold. For example, the calculated differences are ranked from small to large to select the first N differences, and the candidate recall samples corresponding to the candidate recall feature vectors corresponding to these differences are determined as the target retrieval samples.

[0147] In the above-mentioned sample retrieval method, since the target feature extraction model is trained by the classification loss, the target feature extraction model finally obtained can learn features that tend to the label semantic vector, so that the semantic space where the label semantic vector is located can be simulated in the vector space where the training feature vector is located, so that the extracted feature vector can store feature information representing the semantics, thereby improving the accuracy of feature extraction. Moreover, since the label semantic vector includes at least two activated label vector components, the training category label can be more accurately represented. The expected semantic vector is determined based on the label semantic vector and includes expected activation vector components corresponding to the position distribution of at least two activated label vector components, which can make the calculated classification loss more accurate, further improving the accuracy of feature extraction, and then using the feature vector extracted by the target feature extraction model for sample retrieval can improve the accuracy of sample retrieval.

[0148] In one embodiment, the sample retrieval method further includes: performing feature clustering on the candidate recall feature vectors of each candidate recall sample in the candidate recall sample set to obtain a plurality of cluster clusters; each cluster cluster has a corresponding cluster center; for each cluster center, establishing an association relationship between the cluster center and each candidate recall feature vector in the same cluster cluster; based on the query feature vector and the candidate recall feature vector, determining a target retrieval sample corresponding to the query sample from the candidate recall sample set, including: determining a target cluster center from each cluster center based on a feature distance between the query feature vector and each cluster center; obtaining each candidate recall feature vector that has an association relationship with the target cluster center, and determining a target retrieval sample from each obtained candidate recall feature vector based on a feature distance between the query feature vector and each obtained candidate recall feature vector.

[0149] Specifically, after clustering, the computer device establishes the association between each cluster center and each candidate recall feature vector in the same cluster cluster, so that the cluster center can be used as the index of the cluster cluster, so that when searching, the feature distance between the query feature vector and these indexes can be calculated first, so as to screen out the clusters where similar samples are located, and search from the screened cluster clusters to obtain the target search sample. The index query method greatly reduces the amount of calculation in the search process and improves the search efficiency.

[0150] For example, assuming that the candidate recall sample set includes 1,000 candidate recall samples, and after clustering these candidate recall samples, 10 clusters are obtained. Then, when performing retrieval, the computer device can calculate the feature clustering between the query feature vector and the cluster centers of these 10 clusters, select the top 3 clusters with the smallest feature distance, and calculate the feature clustering between the query feature vector and each candidate recall feature vector in these 3 clusters, so as to retrieve the target retrieval sample, and no calculation is required for the remaining 7 clusters.

[0151] In one embodiment, reference Figure 7 , which is a schematic diagram of an index retrieval system in an embodiment. Figure 7 In (a), for all stock images, the target feature extraction model trained by the embodiment of the present application is used to extract image feature vectors, and clustering is performed to obtain multiple clusters, and the clusters are associated with the cluster centers to construct an index system. Figure 7 As shown in Figure (b), in the specific retrieval process, for the query sample, the query feature vector is first extracted through the target feature extraction model, and then compared with the cluster centers of each cluster to determine the target cluster. Then, the target retrieval sample is recalled from the target cluster through the indexing system and sorted and returned.

[0152] In one embodiment, the above-mentioned sample retrieval method also includes a processing step of a feature extraction model, which specifically includes: extracting features from training samples through a feature extraction model to be trained to obtain a training feature vector; performing classification prediction on the training samples based on the training feature vector to obtain a predicted semantic vector; obtaining a label semantic vector of a training category label corresponding to the training sample, the label semantic vector including at least two activated label vector components; determining an expected semantic vector of the training sample based on the label semantic vector, the expected semantic vector including expected activation vector components corresponding to the position distribution of at least two activated label vector components; determining a classification loss based on the difference between each expected activation vector component in the expected semantic vector and the predicted vector components at their corresponding positions in the predicted semantic vector; training the feature extraction model based on the classification loss, and obtaining a target feature extraction model when a training stop condition is met; the target feature extraction model is used to extract a sample feature vector of an input sample.

[0153] The present application also provides an application scenario, which can be applied to the above-mentioned feature extraction model processing method and sample retrieval method. In this application scenario, the content is an image, the training sample is an image sample, and the feature extraction model processing method and sample retrieval method can be applied to image duplicate elimination retrieval.

[0154] In related technologies, image duplicate removal and retrieval use metric learning plus relaxed constraints of symbolic quantization to generate target hash features, and recall similar samples through target hash features. However: 1) Such hash features are constrained by training samples and cannot perceive the problem of insufficient hash coding due to missing data samples and difficulty in representing out-of-domain samples other than training samples. 2) It is not conducive to subsequent parsing and analysis: Since hash features are learned randomly, each hash feature will have redundant bits to varying degrees (for example, an important feature may eventually be represented by different numbers of hash bits such as 0, 1, 2, etc.), which brings great inconvenience to the subsequent model difficult sample analysis. If hash features can effectively represent all possible sample spaces, based on the perception of the overall space, the overall space constrained hash distribution is adopted, and then sufficient hash learning is performed on the target sample space, the main problems of the above hash features can be greatly alleviated.

[0155] Based on this, the present application provides a feature extraction model processing method, which first uses rich multi-labels to establish an overall data space, and maximizes the classification representation distance between each multi-label category so that the classification target is evenly distributed in the semantic space; at the same time, the sample hash feature space is directly associated with the semantic space of its classification, so that the hash feature can further learn the feature measurement capabilities required for duplicate removal and retrieval in the uniform semantic space, so that the hash representation is sufficiently dispersed to support the needs of maximizing the multi-label category representation, thereby avoiding the original hash representation initialized from the random distribution from being overly concentrated in certain hash bits, or the extracted features from being overly concentrated in certain image textures; it also avoids the isolation of the hash feature space and the semantic space. Through the iterative update of the model, it is finally achieved that the hash feature is affected by the uniform spatial representation of the classification, and the representation ability of the multi-bit hash is improved.

[0156] Specifically, the following description is made by taking the method executed by a computer device as an example. It can be understood that the computer device can be Figure 1 The terminal 102 shown may also be a server 104, or a system composed of the terminal 102 and the server 104. The application of the above feature extraction model processing method and sample retrieval method in this application scenario is as follows:

[0157] 1. Data Preparation

[0158] 1. Collect similar sample pairs. Two frames of images can be randomly selected from the image database to form an image pair. By manually marking whether they are similar images, similar images can be grouped into similar sample pairs.

[0159] 2. Triplet mining: Take similar sample pairs as input, and in each batch (the whole data is iterated in epochs in training. In each iteration, each bs pair of all N similarity sample pairs is considered as a batch, with a total of N / bs batches) containing bs sample pairs, perform the following mining to obtain triples: for any image x in a sample pair, calculate the embedding of the sample pair from the remaining bs-1 sample pairs (one image is randomly selected for each pair), calculate the distance between the embedding and x, and sort by distance. Sort from small to large, remove the first K% samples (such as 3%, i.e. (2*bs-2)*3 / 100, such as for bs=64, it is the first 4), take the first 10 samples as negative samples, and form triplets with the positive samples in x respectively. Therefore, each sample generates 10 triplets, and the whole batch obtains 10*bs triplets, where the image x in each triplet is recorded as anchor (a), the image that is a similar sample to x is positive (p), the mined negative sample is negative (n), and the triplet is also recorded as (apn).

[0160] In this embodiment, it is necessary to ensure that the extracted feature vectors can be predicted to be the same (the metric distance is as small as possible) for the same image (completely the same, extremely similar images), and the distance is as large as possible for different images (including images that are only slightly similar but not similar enough, and dissimilar) (it is also necessary to satisfy the order preservation effect: the more dissimilar, the farther the distance). In addition to similar images such as the front and back frames under the same lens of the video, and the attack methods such as adding hue changes to the image, the probability that most images are similar to each other is very low. Therefore, in this embodiment, the positive sample pairs of each batch (sampled from the full amount of samples, which can be understood as the two sample pairs in the batch are mostly different in composition, but there are still a small number of the same) are called noise. For each split-lens 3 similar sample pairs, these 3 are similar to each other. If a batch extracts two of the sample pairs, there must be noise. This type of similar sample pair weak supervision data is processed as follows in this embodiment: remove the most similar small number of samples (the first K%, it is believed that there may be the same samples in the samples here, and should not be used as negative samples for learning), and the rest are valid negative samples. The 10 negative samples with the smallest distance are selected as difficult negative samples. K is a controllable value. The greater the noise in the training sample set, the larger k is.

[0161] 3. Multi-label annotation: For the images of the above similar sample pairs, annotate multiple labels for each image (i.e., the training category labels mentioned above). The multi-labels corresponding to the training sample set (i.e., the specific categories of labels involved) can refer to the large-scale open source imagenet or open-image label system. You can manually annotate the image multi-labels, or use a multi-label model to predict the image labels: For example, use the open source resnet101 network structure to assist the imagenet pre-trained multi-label model to generate multi-label prediction results for the images in the above training set.

[0162] 4. For each image, generate a joint multi-label Hadamard vector representation under multi-label annotation (i.e., the expected semantic vector mentioned above). According to the target hash feature bit Nh to be learned, set the order n of the Hadamard matrix to ensure that n*P=Nh and n>Nclass, that is, the order of the hadamard matrix is ​​less than or equal to the hash bit length, greater than or equal to half the hash bit length, and greater than or equal to the number of multi-label categories. Considering that the hash feature needs to represent the entire image, and the classification information is only a highly abstract image, the length of n should be less than or equal to the hash length; considering that the representation of the entire image needs to introduce image-specific information in addition to the classification representation of the image to distinguish different images of the same classification, P>=1 times the Hadamard order (n) is reserved as a supplement to the classification information. Where Nclass is the number of all labels involved in the training sample set.

[0163] For each label involved in the training sample set, one is selected from the vectors of the other rows in the Hadamard matrix except the first row as its Hadamard vector representation (i.e., the label semantic vector mentioned above). Since one Hadamard vector can only represent one label, if there are multiple labels in the image, the Hadamard vectors of each label can be fused to obtain the joint multi-label Hadamard vector representation of the image. Specifically, it can be determined whether there are activation values ​​in the vector components belonging to the same sorting position in the multiple Hadamard vectors. If there are activation values, the sorting position is treated as an activation value. If there are no activation values, the sorting position is treated as an inactive value. Finally, the activation values ​​and inactive values ​​are arranged according to their sorting positions to form the joint multi-label Hadamard vector representation of the image.

[0164] It can be understood that if an image has only one label, the Hadamard vector of the label is the expected semantic vector of the image.

[0165] 2. Training Process

[0166] The model to be trained includes three parts: basic feature extraction module, hash quantization module, and classification model. The basic feature extraction model and hash quantization module together constitute the feature extraction model. The basic feature extraction model uses resnet101, and the parameters are shown in Table 1. It includes 6 parts, including convolution layer 1, convolution layer 2-convolution layer 5, and pooling layer. Convolution layer 1 is a 7×7×64 convolution with a stride of 2. Convolution layer 2 includes a 3×3 maximum pooling layer and 3 ResNet blocks. Convolution layer 3-convolution layer 5 include 3 ResNet modules, 4 ResNet modules, 23 ResNet modules, and 3 ResNet modules, respectively.

[0167] Table 1

[0168]

[0169]

[0170] The parameters of the hash quantization module are shown in Table 2, including a fully connected layer, which takes the output of the maximum pooling layer as input and outputs a 1x256 floating point. The floating point vector can be mapped to a binary vector (0 or 1) through the sign function, which is the hash feature in the final application. The parameters of the classification model are shown in Table 3, including a classification layer (Fc_class), the input of which is the output of the hash feature generation layer, and the output of which is the n-dimensional prediction probability. Among them, n is the same as the order of the Hadamard matrix.

[0171] Table 2

[0172] The name of the layer Output size layer Hash Quantization Layer 1x256 Fully connected layer

[0173] Table 3

[0174] The name of the layer Output size layer Classification layer 1xn Fully connected layer

[0175] It should be noted that the basic feature extraction model, hash quantization module and classification model can also use other model structures, such as the basic feature extraction model using resnet18CNN, and the hash quantization module using multi-layer fully connected layers. The specific training process is as follows:

[0176] 1. Parameter initialization:

[0177] In the pre-training phase, convolutional layers 1 to 5 use the parameters of ResNet101 pre-trained on the ImageNet dataset. Newly added layers such as feature layers are initialized with a Gaussian distribution with a variance of 0.01 and a mean of 0. The hash quantization layer and classification layer are initialized with a random normal distribution with a variance of 0.01 and a mean of 0.

[0178] 2. Set learning parameters: All parameters in Table 1, Table 2 and Table 3 need to be learned.

[0179] 3. Set the learning rate: lr = 0.0005 learning rate is used for the basic feature extraction model and hash quantization module, and 0.005 learning rate is used for the classification model. After every 10 iterations, lr becomes 0.1 times of the original. When the gradient is fed back, the semantic loss (i.e., classification loss) first updates the parameters of the classification model (i.e., Table 3), and then feeds back to the hash quantization module (Table 2) and the basic feature extraction model (Table 1). Here, setting the learning rate of the hash feature smaller than that of the semantic label can avoid the multi-label semantic loss from being fully transferred to the quantized hash layer, causing excessive semantic learning to affect the measurement effect of the hash quantization module, and causing redundancy in the hash quantization module.

[0180] 4. Learning process: For all data, perform epoch iterations; process all samples once in each iteration until the average epoch loss no longer decreases at a certain epoch, and obtain the image feature extraction model and the target multi-label classification model. The specific operations in each iteration of each epoch are as follows: N similar sample pairs of the full image, each bs sample pair is a batch, divided into Nb = N / bs batches, for each batch, the triples are obtained through the above triple mining, and the following are performed:

[0181] 4.1. Forward calculation: Set all parameters to the state that needs to be learned. During training, a forward calculation is performed on an input image to obtain the hash feature and the n-dimensional probability vector output by the classification layer, represented by Q and P, where Q is a 1x256 vector representing the hash feature, and P is a 1*n-dimensional probability vector, where each bit represents the predicted probability of the feature bit.

[0182] 4.2. Loss calculation: 1) For each triplet, calculate the quantization loss and metric loss; 2) For each image in the bs pair sample pair, obtain the expected semantic vector of the image respectively, and calculate the multi-label cross entropy loss function between the n-dimensional probability vector output by the classification layer and the expected semantic vector; 3) For the bs pair sample pair, calculate the L2 loss of the effective bit of the Hadamard multi-label and the Hadamard multi-label target in its hash output result respectively, so that the Hadamard multi-label bit in the sample hash bit is consistent with the Hadamard label of the sample, thereby simulating the Hadamard space in the hash space. Calculate the sum of the three losses as the total loss. The details are as follows:

[0183] For each sample pair in the batch (a total of bs sample pairs), the hash loss is the average of the hash losses of the 10 mined triples, the average of the Hadamard classification losses of the two image samples, and the regression loss (L2 loss) of the hash representation of the two image samples consistent with the Hadamard representation. The classification loss is the average of the classification losses of the two images. For details, refer to the following formula (4):

[0184]

[0185] Among them, w1=0.1, w3=0.2, w2=1-w1-w3. The effect of adding multi-label semantic space learning based on Hadamard vector in this embodiment is: adding uniform multi-activated Hadamard vector representation on top of hash feature can constrain the hash feature to have a uniform distribution. Except for the first row of Hadamard vector, the sum of other rows is 0, that is, the number of -1 and 1 is the same, which can be understood as the representation of the category activation bit is uniform compared to the one-hot representation and always 1. The activation degree may be determined according to the number of activated label vector components under the same feature position in multiple semantic label vectors. For details, please refer to the description in the above embodiment.

[0186] In this embodiment, considering that metric learning converges slowly - finding similar and dissimilar parts on two images is much more difficult than finding commonalities of the same category under multiple images of the same category, semantic learning can accelerate convergence and avoid missing recall of semantically related images due to improper local representation in metric learning, and W1 = 0.1 is set. However, fast semantic convergence is also prone to overfitting of quantitative features, which is constrained by the quantitative learning rate being 0.1 times that of semantic learning and the semantic multi-label loss weight w1 being 1 / 9 of that of metric learning w2.

[0187] Setting W3 = 0.2 can effectively ensure that the metric learning sample pairs are close to each other, and then use smaller weights to adjust the sample pairs to the vicinity of the semantic representation, avoiding the loss or reduction of the metric learning effect due to the adjustment of semantic loss.

[0188] The following is a detailed introduction to each loss:

[0189] 1) L hash : By L coding With L triplet For details, refer to the following formula (5):

[0190] L hash =w 21 L triplet +w 22 L coding (5)

[0191] Among them, L tripletRefer to the above formula (1) for calculation, L coding The calculation can be performed with reference to formulas (2) and (3) in the above embodiments. 21 Can be set to 1, w 22 It can be set to 0.5. reg Convergence ratio L triplet Quick, to ensure L triplet It plays a dominant role in the overall loss, thus ensuring that embedding always has the ability to measure similarity. 22 Set to 0.5 (or other values ​​less than 1, which can be adjusted as needed). It can be understood that in practical applications, the sign function can be directly used to generate a quantized binary vector for the output Q, which can be used for image retrieval.

[0192] 2) L class : For the probability vector output by the classification model, the multi-label loss between it and the expected semantic vector of the image can be calculated by referring to the following formula (6), where t[i] is a 1*n vector of 0 and 1 (where -1 in the expected semantic vector obtained by the Hadamard vector is replaced by 0), and o[i] is the predicted probability at each of the 1*n Hadamard positions (i.e., the sorting position in the Hadamard vector).

[0193]

[0194] 3) L reg :The regression loss Lreg of semantic representation modeling in the hash space of sample pairs is as follows, where a and p represent two samples in the sample pair, and Hash a Represents the hash output of hash sample a, Hada a To represent the Hadamard multi-label target of sample a, refer to the following formula (7):

[0195] L reg =(Hash a -Hada a ) 2 +(Hash p -Hada p ) 2 (7)

[0196] When P=1, Nh=n, the dimension of the expected semantic vector and the training feature vector is the same, and the minimum L2 distance can be directly used as the constraint of the modeling semantic space. It can be understood that in this embodiment, the training feature vector is a hash feature, so the sorting position in the training feature vector is the hash position.

[0197] When P>1, that is, Nh is P times n, formula (7) needs to be adjusted as follows: only keep the bits in the hash bit that are mapped to the Hadamard bit consistent. The relationship between the hash bit and the Hadamard bit mapping can be referred to Figure 3 , one can be selected from each P hash bits to map to the Hadamard bit, where P is a multiple of the P bit length in Nh=n*P. For each Hadamard bit i (a total of n bits) of image j, the regression loss between the Hadamard bit and the corresponding hash bit (Pi+q, q ​​represents the offset) is minimized. For details, please refer to the following formula (8):

[0198]

[0199] 4.3. Model parameter update: Using the stochastic gradient descent algorithm, the target model loss value obtained in the previous step is subjected to gradient backward calculation to obtain the updated value of all model parameters, and the updated value is used to update the parameters of all models.

[0200] 3. Application of Quantitative Feature Retrieval

[0201] For all stock images, all Qs are extracted using the trained image feature extraction model, and the binary quantized features obtained by activation with the sign function are stored in the library. For the query image, Q is extracted using the above image feature extraction model and the binary quantized features are obtained by binary quantization, which are then compared one by one with the binary quantized features of the stock. The Hamming distance calculation for the binary quantized features can speed up the calculation efficiency (compared with the floating-point embedding features). After calculating the distance, the topK most similar ones are sorted from small to large and returned. In this way, stock images with semantic similarity to the query image can be retrieved (i.e., the target retrieval result samples mentioned above).

[0202] The above embodiments have the following beneficial effects:

[0203] On the one hand, with the help of multi-label semantic space constraints with rich samples, hash features can not only represent metric learning features, but also store feature information representing semantics, avoiding the bias of hash feature learning due to sample bias or insufficient features, which makes it impossible to effectively represent out-of-domain samples, thereby improving the recall accuracy in retrieval scenarios containing out-of-domain samples.

[0204] On the other hand, the multi-bit uniform activation of multi-label targets avoids the problems of one-hot multi-label being unable to effectively utilize all bits and being prone to overfitting. Through gradient backpropagation, the hash bits are also concentrated on the features required for specific classification bits, thereby generating problems such as redundant features.

[0205] In addition, with the help of Hadamard-based hash code metric constraints, the overall distribution of samples in the hash space simulates the uniformly distributed semantic space (simulating the semantic space in the non-semantic space and fine-tuning the learning of the hash representation of the non-semantic space). On the one hand, the semantic space greatly enhances the multi-bit activation effect of the hash feature, and achieves the effect of hash feature metric representation of the image by fine-tuning the non-semantic bits in the hash feature; on the other hand, the non-semantic space with less information can also obtain more meaningful supervision information from the semantic space to obtain a more uniform non-semantic hash distribution, avoiding the uneven distribution of non-semantic hashes, and thus the problem of poor retrieval effect due to excessive concentration on certain features due to low discrimination of these features in retrieval.

[0206] The present application also provides another application scenario, which is applicable to the above-mentioned feature extraction model processing method. In this application scenario, the image feature extraction model and the target multi-label classification model are trained through the same steps as the above application scenario, and the image feature extraction model and the target multi-label classification model are spliced ​​to obtain the target classification model. According to the target classification model, multi-label classification of the input image can be realized.

[0207] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0208] Based on the same inventive concept, the embodiment of the present application also provides a feature extraction model processing device for implementing the feature extraction model processing method involved above and a sample retrieval device for implementing the sample retrieval method involved above. The implementation solutions to the problems provided by these devices are similar to the implementation solutions recorded in the above methods, so the specific limitations in one or more feature extraction model processing device embodiments and sample retrieval device embodiments provided below can refer to the limitations on the feature extraction model processing method above, and will not be repeated here.

[0209] In one embodiment, Figure 8 As shown, a feature extraction model processing device 800 is provided, comprising:

[0210] A feature extraction module 802 is used to extract features from training samples using a feature extraction model to be trained to obtain a training feature vector;

[0211] A classification prediction module 804 is used to perform classification prediction on the training samples based on the training feature vector to obtain a predicted semantic vector;

[0212] A label vector acquisition module 806 is used to acquire a label semantic vector of a training category label corresponding to the training sample, where the label semantic vector includes at least two activated label vector components;

[0213] An expected vector determination module 808 is used to determine an expected semantic vector of the training sample based on the label semantic vector, wherein the expected semantic vector includes an expected activation vector component corresponding to the position distribution of at least two activation label vector components;

[0214] A classification loss determination module 810, configured to determine a classification loss based on a difference between each expected activation vector component in the expected semantic vector and a predicted vector component at each corresponding position in the predicted semantic vector;

[0215] The model training module 812 is used to train the feature extraction model based on the classification loss, and when the training stop condition is met, a target feature extraction model is obtained; the target feature extraction model is used to extract the sample feature vector of the input sample.

[0216] The feature extraction model processing device performs feature extraction on the training sample through the feature extraction model to be trained to obtain the training feature vector, performs classification prediction on the training sample based on the training feature vector to obtain the predicted semantic vector, obtains the label semantic vector of the training category label corresponding to the training sample, determines the expected semantic vector of the training sample based on the label semantic vector, and then determines the classification loss based on the difference between each expected activation vector component in the expected semantic vector and the predicted vector component at the corresponding position in the predicted semantic vector, trains the feature extraction model through the classification loss, and finally obtains the target feature extraction model that can learn the features that tend to the label semantic vector, so that the semantic space where the label semantic vector is located can be simulated in the vector space where the training feature vector is located, so that the extracted feature vector can store the feature information representing the semantics, thereby improving the accuracy of feature extraction, and because the label semantic vector includes at least two activated label vector components, the training category label can be more accurately represented, and the expected semantic vector is determined based on the label semantic vector and includes the expected activation vector component corresponding to the position distribution of at least two activated label vector components, which can make the calculated classification loss more accurate, further improving the accuracy of feature extraction.

[0217] In one embodiment, the expected vector determination module is also used to group the label vector components with the same sorting position in each label semantic vector into a feature component set; for a feature component set that includes an activated label vector component, the expected vector component at the sorting position to which the feature component set belongs is set to an activated value; for a feature component set that does not include an activated label vector component, the expected vector component at the sorting position to which the feature component set belongs is set to an inactive value; and each expected vector component is combined according to its corresponding sorting position to form an expected semantic vector of the training sample.

[0218] In one embodiment, a classification loss determination module is used to determine, for each expected vector component in the expected semantic vector, an initial value of the classification loss component at the sorting position of the expected vector component based on the difference between the expected vector component and the predicted vector component corresponding to the sorting position in the predicted semantic vector; to count the activated label vector components in the feature component set at the sorting position of the expected vector component, and to determine the activation degree of the sorting position of the expected vector component based on the statistical results; to weight the initial value of the classification loss component based on the activation degree to obtain a target value of the classification loss component; and to count the target values ​​of each classification loss component to determine the classification loss.

[0219] In one embodiment, the above-mentioned device also includes: a semantic constraint loss determination module, which is used to determine the semantic constraint loss based on the difference between the training feature vector of the training sample and the expected semantic vector, and the semantic constraint loss is used to semantically constrain the training feature vector extracted by the feature extraction model; a classification loss determination module, which is also used to determine the target loss based on the semantic constraint loss and the classification loss; the feature extraction model is trained based on the target loss, and when the training stop condition is met, the target feature extraction model is obtained.

[0220] In one embodiment, the semantic constraint loss determination module is also used to calculate the vector distance between the training feature vector and the expected semantic vector if the dimension of the training feature vector is consistent with the dimension of the expected semantic vector, and use the vector distance as the semantic constraint loss corresponding to the training sample.

[0221] In one embodiment, the semantic constraint loss determination module is also used to select a training vector component from the training feature vector for each expected vector component in the expected semantic vector if the dimension of the training feature vector is greater than the dimension of the expected semantic vector, and use the selected training vector component as the mapping vector component of the expected vector component; calculate the difference between the expected vector component and the mapping vector component to obtain the semantic constraint loss component corresponding to the expected vector component; and count the various semantic constraint loss components to obtain the semantic constraint loss corresponding to the training sample.

[0222] In one embodiment, the semantic constraint loss determination module is also used to group the training vector components in the training feature vector according to the dimension of the expected semantic vector to obtain multiple vector component groups whose number is consistent with the dimension of the expected semantic vector; for each expected vector component in the expected semantic vector, select a vector component group from the multiple feature component groups as the target vector component group; select a training vector component from the target vector component group as the mapping vector component corresponding to the expected vector component.

[0223] In one embodiment, the training samples belong to a training sample set, and the training category labels corresponding to the training samples in the training sample set constitute a label set. The above-mentioned device also includes a label vector generation module for constructing a target semantic space; the target vector dimension of the target semantic space does not exceed the vector dimension of the training feature vector; each training category label in the label set is mapped to the target semantic space respectively to obtain the label semantic vector of each training category label.

[0224] In one embodiment, a label vector generation module is used to generate a target matrix whose matrix order matches the target vector dimension, wherein the target matrix includes multiple candidate representation vectors, and the multiple candidate representation vectors form a target semantic space; the candidate representation vectors include a first numerical value and a second numerical value of equal quantity; for each semantic category label in the semantic category label set, a candidate representation vector is selected from the multiple candidate representation vectors as the label semantic vector corresponding to each semantic category label.

[0225] In one embodiment, the classification loss determination module is also used to obtain a comparison feature vector corresponding to a comparison training sample of the training sample, and obtain a feature extraction loss based on the difference between the training feature vector and the comparison feature vector; obtain a target loss based on the feature extraction loss and the classification loss; adjust the parameters of the feature extraction model based on the target loss and continue training, and obtain a target feature extraction model when the training stop condition is met.

[0226] In one embodiment, the comparison feature vector includes a positive comparison feature vector corresponding to a positive comparison training sample and a negative comparison feature vector corresponding to a negative comparison training sample; the classification loss determination module is further used to obtain a positive feature difference value, which is a feature difference value between the training feature vector and the positive comparison feature vector; obtain a negative feature difference value, which is a feature difference value between the training feature vector and the negative comparison feature vector; and determine the feature extraction loss based on the positive feature difference value and the negative feature difference value.

[0227] In one embodiment, the feature extraction module is also used to extract the initial sample features of the training sample through the feature extraction model, and quantize the initial sample features to obtain the training feature vector of the training sample; the classification loss determination module is also used to determine the quantization target corresponding to each quantization value in the training feature vector based on a preset sign function, and determine the quantization loss based on the difference between each quantization value and the corresponding quantization target; obtain the target loss based on the quantization loss and the classification loss; adjust the parameters of the feature extraction model based on the target loss and continue training, and obtain the target feature extraction model when the training stop condition is met.

[0228] In one embodiment, Fig. 9 As shown, a sample retrieval device 900 is provided, comprising:

[0229] The sample acquisition module 902 is used to acquire a query sample and a candidate recall sample set;

[0230] The feature extraction module 904 is used to input the query sample and the candidate recall sample in the candidate recall sample set into the target feature extraction model respectively, and obtain the query feature vector corresponding to the query sample and the candidate recall feature vector corresponding to the candidate recall sample; wherein the target feature extraction model is obtained by training the feature extraction model to be trained through the classification loss, the classification loss is determined based on the difference between each expected activation vector component in the expected semantic vector and the predicted vector component at the corresponding position in the predicted semantic vector, the predicted semantic vector is obtained by classifying and predicting the training sample based on the training feature vector, the expected semantic vector is determined based on the label semantic vector of the training category label corresponding to the training sample, the label semantic vector includes at least two activation label vector components, the expected semantic vector includes the expected activation vector component corresponding to the position distribution of at least two activation label vector components, and the training feature vector is obtained by extracting features from the training sample by the feature extraction model to be trained;

[0231] The retrieval module 906 is used to determine a target retrieval sample corresponding to the query sample from the candidate recall sample set based on the query feature vector and the candidate recall feature vector.

[0232] The above-mentioned sample retrieval device, since the target feature extraction model is trained by the classification loss, the target feature extraction model finally obtained can learn the features that tend to the label semantic vector, so that the semantic space where the label semantic vector is located can be simulated in the vector space where the training feature vector is located, so that the extracted feature vector can store feature information representing the semantics, thereby improving the accuracy of feature extraction, and since the label semantic vector includes at least two activated label vector components, the training category label can be more accurately represented, and the expected semantic vector is determined based on the label semantic vector and includes the expected activation vector component corresponding to the position distribution of at least two activated label vector components, which can make the calculated classification loss more accurate, further improving the accuracy of feature extraction, and then using the feature vector extracted by the target feature extraction model for sample retrieval, which can improve the accuracy of sample retrieval.

[0233] In one embodiment, the above-mentioned device also includes: an association relationship establishment module, which is used to perform feature clustering on the candidate recall feature vectors of each candidate recall sample in the candidate recall sample set to obtain multiple cluster clusters; each cluster cluster has a corresponding cluster center; for each cluster center, an association relationship is established between the cluster center and each candidate recall feature vector in the same cluster cluster; a retrieval module, which is used to determine the target cluster center from each cluster center based on the feature distance between the query feature vector and each cluster center; obtain each candidate recall feature vector that has an association relationship with the target cluster center, and determine the target retrieval sample from each candidate recall feature vector obtained based on the feature distance between the query feature vector and each obtained candidate recall feature vector.

[0234] Each module in the above-mentioned feature extraction model processing device and sample retrieval device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0235] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store training sample data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a feature extraction model processing method or a sample retrieval method is implemented.

[0236] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a feature extraction model processing method or a sample retrieval method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0237] Those skilled in the art will understand that Fig.10 and Fig.11The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0238] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned feature extraction model method or sample retrieval steps when executing the computer program.

[0239] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned feature extraction model method or sample retrieval steps are implemented.

[0240] In one embodiment, a computer program product is provided, including a computer program, which implements the above-mentioned feature extraction model method or sample retrieval steps when executed by a processor.

[0241] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0242] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0243] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0244] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A feature extraction model processing method, characterized in that: The method comprises: Extracting features from training samples using a feature extraction model to be trained to obtain a training feature vector; the training sample is any one of text, audio or image; Performing classification prediction on the training samples based on the training feature vector to obtain a predicted semantic vector; Obtaining a label semantic vector of a training category label corresponding to the training sample, wherein the label semantic vector includes at least two activated label vector components; the activated label vector component is an activated label vector component in the label semantic vector, and the label vector component is a vector component contained in the label semantic vector; Based on the label semantic vector, determining the expected semantic vector of the training sample, wherein the expected semantic vector includes an expected activation vector component corresponding to the position distribution of the at least two activated label vector components; the expected activation vector component is an activated expected vector component, and the expected vector component is a vector component included in the expected semantic vector; Determining a classification loss based on a difference between each of the expected activation vector components in the expected semantic vector and the predicted vector components at the respective corresponding positions in the predicted semantic vector; The feature extraction model is trained based on the classification loss, and when a training stop condition is met, a target feature extraction model is obtained; the target feature extraction model is used to extract a sample feature vector of an input sample.

2. The method according to claim 1, characterized in that The training sample corresponds to a plurality of training category labels, and determining the expected semantic vector of the training sample based on the label semantic vector includes: The label vector components with the same sorting position in each label semantic vector are combined into a feature component set; For a feature component set including an activated label vector component, setting the expected vector component at the sorted position to which the feature component set belongs as an activation value; For a feature component set that does not include an activated label vector component, setting the expected vector component at the sorting position to which the feature component set belongs to an inactive value; The expected vector components are combined according to their corresponding sorting positions to form the expected semantic vector of the training sample.

3. The method according to claim 2, characterized in that The determining of the classification loss based on the difference between each of the expected activation vector components in the expected semantic vector and the predicted vector components at the corresponding positions in the predicted semantic vector comprises: For each expected vector component in the expected semantic vector, determining an initial value of a classification loss component at a sorted position where the expected vector component is located based on a difference between the expected vector component and a predicted vector component corresponding to a sorted position in the predicted semantic vector; Performing statistics on the activated label vector components in the feature component set at the sorting position where the expected vector component is located, and determining the activation degree of the sorting position where the expected vector component is located based on the statistical results; Performing weighted processing on the initial value of the classification loss component based on the activation degree to obtain a target value of the classification loss component; The target values ​​of each classification loss component are counted to determine the classification loss.

4. The method according to claim 1, characterized in that: Before training the feature extraction model based on the classification loss and obtaining the target feature extraction model when a training stop condition is met, the method further includes: Determining a semantic constraint loss based on a difference between the training feature vector of the training sample and the expected semantic vector, wherein the semantic constraint loss is used to semantically constrain the training feature vector extracted by the feature extraction model; The feature extraction model is trained based on the classification loss, and when a training stop condition is met, a target feature extraction model is obtained, including: Determining a target loss based on the semantic constraint loss and the classification loss; The feature extraction model is trained based on the target loss, and when a training stop condition is met, a target feature extraction model is obtained.

5. The method according to claim 4, characterized in that The determining of the semantic constraint loss based on the difference between the training feature vector of the training sample and the expected semantic vector comprises: If the dimension of the training feature vector is consistent with the dimension of the expected semantic vector, then A vector distance between the training feature vector and the expected semantic vector is calculated, and the vector distance is used as the semantic constraint loss corresponding to the training sample.

6. The method according to claim 4, characterized in that The determining of the semantic constraint loss based on the difference between the training feature vector of the training sample and the expected semantic vector comprises: If the dimension of the training feature vector is greater than the dimension of the expected semantic vector, then For each expected vector component in the expected semantic vector, select a training vector component from the training feature vector, and use the selected training vector component as a mapping vector component of the expected vector component; Calculating the difference between the expected vector component and the mapped vector component to obtain a semantic constraint loss component corresponding to the expected vector component; Each semantic constraint loss component is counted to obtain the semantic constraint loss corresponding to the training sample.

7. The method according to claim 6, characterized in that Selecting a training vector component from the training feature vector and using the selected training vector component as a mapping vector component of the expected vector component includes: According to the dimension of the expected semantic vector, the training vector components in the training feature vector are grouped to obtain a plurality of vector component groups whose number is consistent with the dimension of the expected semantic vector; For each expected vector component in the expected semantic vector, selecting a vector component group from the plurality of vector component groups as a target vector component group; A training vector component is selected from the target vector component group as a mapping vector component corresponding to the expected vector component.

8. The method according to claim 1, characterized in that The training samples belong to a training sample set, and the training category labels corresponding to the training samples in the training sample set form a label set. The method further includes: Constructing a target semantic space; the target vector dimension of the target semantic space does not exceed the vector dimension of the training feature vector; Each training category label in the label set is mapped to the target semantic space to obtain a label semantic vector of each training category label.

9. The method according to claim 8, characterized in that The constructing of the target semantic space includes: Generate a target matrix whose matrix order matches the dimension of the target vector, wherein the target matrix includes a plurality of candidate representation vectors, and the plurality of candidate representation vectors form the target semantic space; the candidate representation vectors include a first value and a second value of equal quantity; The step of mapping each training category label in the label set to the target semantic space to obtain a label semantic vector of each training category label includes: For each training category label in the label set, a candidate representation vector is selected from the multiple candidate representation vectors as the label semantic vector corresponding to each training category label.

10. The method according to claim 1, characterized in that The feature extraction model is trained based on the classification loss, and when a training stop condition is met, a target feature extraction model is obtained, including: Obtaining a comparison feature vector corresponding to a comparison training sample of the training sample, and obtaining a feature extraction loss based on a difference between the training feature vector and the comparison feature vector; Obtaining a target loss based on the feature extraction loss and the classification loss; The parameters of the feature extraction model are adjusted based on the target loss and training is continued. When the training stop condition is met, a target feature extraction model is obtained.

11. The method according to claim 10, characterized in that The contrast feature vector includes a positive contrast feature vector corresponding to a positive contrast training sample and a negative contrast feature vector corresponding to a negative contrast training sample; and the feature extraction loss obtained based on the difference between the training feature vector and the contrast feature vector includes: Obtaining a positive feature difference value, where the positive feature difference value is a feature difference value between the training feature vector and the positive comparison feature vector; Obtain a negative feature difference value, where the negative feature difference value is a feature difference value between the training feature vector and the negative comparison feature vector; A feature extraction loss is determined based on the positive feature difference value and the negative feature difference value.

12. The method according to claim 1, characterized in that The step of extracting features from the training samples using the feature extraction model to be trained to obtain a training feature vector includes: Extracting initial sample features of the training sample through the feature extraction model, and quantizing the initial sample features to obtain a training feature vector of the training sample; The feature extraction model is trained based on the classification loss, and when a training stop condition is met, a target feature extraction model is obtained, including: Determine a quantization target corresponding to each quantization value in the training feature vector based on a preset sign function, and determine a quantization loss based on a difference between each quantization value and the corresponding quantization target; Obtaining a target loss based on the quantization loss and the classification loss; The parameters of the feature extraction model are adjusted based on the target loss and training is continued. When the training stop condition is met, a target feature extraction model is obtained.

13. A sample retrieval method, characterized in that: The method comprises: Obtain query samples and candidate recall sample sets; Inputting the query sample and the candidate recall sample in the candidate recall sample set into the target feature extraction model respectively to obtain a query feature vector corresponding to the query sample and a candidate recall feature vector corresponding to the candidate recall sample; Wherein, the target feature extraction model is obtained by training the feature extraction model to be trained through classification loss, the classification loss is determined based on the difference between each expected activation vector component in the expected semantic vector and the predicted vector component at the corresponding position in the predicted semantic vector, the predicted semantic vector is obtained by classifying and predicting the training sample based on the training feature vector, the expected semantic vector is determined based on the label semantic vector of the training category label corresponding to the training sample, the label semantic vector includes at least two activation label vector components, the expected semantic vector includes expected activation vector components corresponding to the position distribution of the at least two activation label vector components, and the training feature vector is obtained by extracting features from the training sample by the feature extraction model to be trained; Based on the query feature vector and the candidate recall feature vector, determining a target retrieval sample corresponding to the query sample from the candidate recall sample set; Among them, the training sample is any one of text, audio or image; the activated label vector component is the activated label vector component in the label semantic vector, and the label vector component is the vector component contained in the label semantic vector; the expected activation vector component is the activated expected vector component, and the expected vector component is the vector component contained in the expected semantic vector.

14. The method according to claim 13, characterized in that The method further comprises: Performing feature clustering on the candidate recall feature vectors of each candidate recall sample in the candidate recall sample set to obtain a plurality of clusters; each cluster has a corresponding cluster center; For each cluster center, establishing an association relationship between the cluster center and each candidate recall feature vector in the same cluster cluster; The determining, based on the query feature vector and the candidate recall feature vector, a target retrieval sample corresponding to the query sample from the candidate recall sample set comprises: Determining a target cluster center from each of the cluster centers based on the characteristic distance between the query characteristic vector and each of the cluster centers; Each candidate recall feature vector associated with the target cluster center is obtained, and a target retrieval sample is determined based on a feature distance between the query feature vector and each obtained candidate recall feature vector.

15. A feature extraction model processing device, characterized in that: The device comprises: A feature extraction module, used to extract features from training samples through a feature extraction model to be trained to obtain a training feature vector; the training sample is any one of text, audio or image; A classification prediction module, used to perform classification prediction on the training sample based on the training feature vector to obtain a predicted semantic vector; A label vector acquisition module, used to acquire a label semantic vector of a training category label corresponding to the training sample, wherein the label semantic vector includes at least two activated label vector components; the activated label vector component is an activated label vector component in the label semantic vector, and the label vector component is a vector component contained in the label semantic vector; An expected vector determination module is used to determine the expected semantic vector of the training sample based on the label semantic vector, wherein the expected semantic vector includes an expected activation vector component corresponding to the position distribution of the at least two activated label vector components; the expected activation vector component is an activated expected vector component, and the expected vector component is a vector component included in the expected semantic vector; A classification loss determination module, configured to determine a classification loss based on a difference between each of the expected activation vector components in the expected semantic vector and the predicted vector components at the respective corresponding positions in the predicted semantic vector; A model training module is used to train the feature extraction model based on the classification loss, and when the training stop condition is met, a target feature extraction model is obtained; the target feature extraction model is used to extract a sample feature vector of an input sample.

16. The device according to claim 15, characterized in that The expected vector determination module is also used to form a feature component set from label vector components with the same sorting position in each label semantic vector; for a feature component set including an activated label vector component, the expected vector component at the sorting position to which the feature component set belongs is set as an activated value; for a feature component set not including an activated label vector component, the expected vector component at the sorting position to which the feature component set belongs is set as an inactive value; and each expected vector component is combined according to its corresponding sorting position to form the expected semantic vector of the training sample.

17. The device according to claim 16, characterized in that The classification loss determination module is also used to determine, for each expected vector component in the expected semantic vector, an initial value of the classification loss component at the sorting position where the expected vector component is located based on the difference between the expected vector component and the predicted vector component corresponding to the sorting position in the predicted semantic vector; perform statistics on the activated label vector components in the feature component set at the sorting position where the expected vector component is located, and determine the activation degree at the sorting position where the expected vector component is located based on the statistical results; The initial value of the classification loss component is weighted based on the activation degree to obtain a target value of the classification loss component; and the target values ​​of each classification loss component are counted to determine the classification loss.

18. The device according to claim 15, characterized in that The device also includes: A semantic constraint loss determination module, used to determine a semantic constraint loss based on a difference between a training feature vector of the training sample and the expected semantic vector, wherein the semantic constraint loss is used to semantically constrain the training feature vector extracted by the feature extraction model; The classification loss is also used to determine a target loss based on the semantic constraint loss and the classification loss; the feature extraction model is trained based on the target loss, and when a training stop condition is met, a target feature extraction model is obtained.

19. The device according to claim 18, characterized in that The semantic constraint loss determination module is also used to calculate the vector distance between the training feature vector and the expected semantic vector if the dimension of the training feature vector is consistent with the dimension of the expected semantic vector, and use the vector distance as the semantic constraint loss corresponding to the training sample.

20. The device according to claim 18, characterized in that The semantic constraint loss determination module is also used to select a training vector component from the training feature vector for each expected vector component in the expected semantic vector if the dimension of the training feature vector is greater than the dimension of the expected semantic vector, and use the selected training vector component as the mapping vector component of the expected vector component; calculate the difference between the expected vector component and the mapping vector component to obtain the semantic constraint loss component corresponding to the expected vector component; and count each semantic constraint loss component to obtain the semantic constraint loss corresponding to the training sample.

21. The device according to claim 20, characterized in that The semantic constraint loss determination module is also used to group the training vector components in the training feature vector according to the dimension of the expected semantic vector to obtain a plurality of vector component groups whose number is consistent with the dimension of the expected semantic vector; for each expected vector component in the expected semantic vector, select a vector component group from the multiple vector component groups as a target vector component group; and select a training vector component from the target vector component group as a mapping vector component corresponding to the expected vector component.

22. The device according to claim 15, characterized in that The training samples belong to a training sample set, and the training category labels corresponding to the training samples in the training sample set form a label set. The device also includes: A label vector generation module is used to construct a target semantic space; the target vector dimension of the target semantic space does not exceed the vector dimension of the training feature vector; each training category label in the label set is mapped to the target semantic space to obtain a label semantic vector for each training category label.

23. The device according to claim 22, characterized in that The label vector generation module is further used to generate a target matrix whose matrix order matches the dimension of the target vector, wherein the target matrix includes a plurality of candidate representation vectors, and the plurality of candidate representation vectors form the target semantic space; The candidate characterization vector includes equal numbers of first values ​​and second values; The label vector generation module is further used to select a candidate representation vector from the multiple candidate representation vectors for each training category label in the label set as the label semantic vector corresponding to each training category label.

24. The device according to claim 15, characterized in that The classification loss determination module is also used to obtain a comparison feature vector corresponding to a comparison training sample of the training sample, and obtain a feature extraction loss based on the difference between the training feature vector and the comparison feature vector; obtain a target loss based on the feature extraction loss and the classification loss; adjust the parameters of the feature extraction model based on the target loss and continue training, and obtain a target feature extraction model when a training stop condition is met.

25. The device according to claim 24, characterized in that The contrast feature vector includes a positive contrast feature vector corresponding to a positive contrast training sample and a negative contrast feature vector corresponding to a negative contrast training sample; the classification loss determination module is also used to obtain a positive feature difference value, which is a feature difference value between the training feature vector and the positive contrast feature vector; obtain a negative feature difference value, which is a feature difference value between the training feature vector and the negative contrast feature vector; and determine a feature extraction loss based on the positive feature difference value and the negative feature difference value.

26. The device according to claim 15, characterized in that The feature extraction module is also used to extract initial sample features of the training sample through the feature extraction model, and quantize the initial sample features to obtain a training feature vector of the training sample; The classification loss determination module is further used to determine the quantization target corresponding to each quantization value in the training feature vector based on a preset sign function, determine the quantization loss based on the difference between each quantization value and the corresponding quantization target; and obtain the target loss based on the quantization loss and the classification loss; The parameters of the feature extraction model are adjusted based on the target loss and training is continued. When the training stop condition is met, a target feature extraction model is obtained.

27. A sample retrieval device, characterized in that: The device comprises: A sample acquisition module is used to obtain query samples and candidate recall sample sets; A feature extraction module, used for inputting the query sample and the candidate recall sample in the candidate recall sample set into a target feature extraction model respectively, to obtain a query feature vector corresponding to the query sample and a candidate recall feature vector corresponding to the candidate recall sample; wherein the target feature extraction model is obtained by training the feature extraction model to be trained through classification loss, the classification loss is determined based on the difference between each expected activation vector component in the expected semantic vector and the predicted vector component at the corresponding position in the predicted semantic vector, the predicted semantic vector is obtained by classifying and predicting the training sample based on the training feature vector, the expected semantic vector is determined based on the label semantic vector of the training category label corresponding to the training sample, the label semantic vector includes at least two activation label vector components, the expected semantic vector includes an expected activation vector component corresponding to the position distribution of the at least two activation label vector components, and the training feature vector is obtained by extracting features from the training sample by the feature extraction model to be trained; A retrieval module, configured to determine a target retrieval sample corresponding to the query sample from the candidate recall sample set based on the query feature vector and the candidate recall feature vector; Among them, the training sample is any one of text, audio or image; the activated label vector component is the activated label vector component in the label semantic vector, and the label vector component is the vector component contained in the label semantic vector; the expected activation vector component is the activated expected vector component, and the expected vector component is the vector component contained in the expected semantic vector.

28. The device according to claim 27, characterized in that The device also includes: The association relationship establishment module is used to perform feature clustering on the candidate recall feature vectors of each candidate recall sample in the candidate recall sample set to obtain multiple clusters; each cluster has a corresponding cluster center; for each cluster center, an association relationship is established between the cluster center and each candidate recall feature vector in the same cluster cluster; The retrieval module is also used to determine a target cluster center from each cluster center based on the feature distance between the query feature vector and each cluster center; obtain each candidate recall feature vector that has an association relationship with the target cluster center, and determine a target retrieval sample based on the feature distance between the query feature vector and each obtained candidate recall feature vector.

29. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 or 13 to 14 are implemented.

30. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 or 13 to 14 are implemented.

31. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 or 13 to 14 are implemented.

Images