Data processing method, device, electronic device and storage medium

Abstract

This application provides a data processing method, apparatus, electronic device, and storage medium, relating to the fields of artificial intelligence, multimedia, and cloud technology. The method includes: acquiring data to be processed; extracting feature vectors from the data to be processed using a trained feature extraction model; determining the category of the data to be processed based on the feature vectors, or identifying target data matching the data to be processed from the dataset; the feature extraction model is obtained by training a neural network model with multiple samples having at least one labeled label, where the labeled label is the true category label of the sample; during training, the target feature vector of each sample is obtained by fusing the initial feature vector of the sample and the label feature vectors of each labeled label of the sample; the total training loss is determined based on the labeled labels and target feature vectors of each sample. Based on this method, the data processing effect can be effectively improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and multimedia technology, and more specifically, to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the research and development of artificial intelligence technology, it has been studied and applied in many fields. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Through machine learning, neural network models that meet application requirements can be trained.

[0003] Classification models are neural network models commonly used in many fields and application scenarios. Classification models extract features from data (such as images, text, videos, speech, etc.) through feature extraction modules and predict the category of the data based on the extracted data features. The expressive power of the data features extracted by the feature extraction module plays a crucial role in the accuracy of the model's prediction results, especially in multi-label classification scenarios.

[0004] There are various training methods for feature extraction models, but the discriminative power of the features extracted by most of the existing well-trained feature extraction models still needs to be improved, especially for multi-label classification applications. Summary of the Invention

[0005] The purpose of this application is to provide a data processing method, apparatus, electronic device, and storage medium. This method can effectively improve the expressive power of features extracted by a feature extraction model and enhance data processing performance. To achieve the above objective, the technical solutions provided by this application are as follows:

[0006] On one hand, embodiments of this application provide a data processing method, the method comprising:

[0007] Acquire data to be processed, wherein the data to be processed is data containing information of at least one modality in multimedia information;

[0008] The feature vector of the data to be processed is extracted using a trained feature extraction model;

[0009] Based on the feature vector of the data to be processed, determine the category of the data to be processed, or identify target data that matches the data to be processed from the dataset;

[0010] The feature extraction model is obtained by repeatedly training a neural network model on multiple samples with at least one labeled label. The labeled label is the true class label of the sample, and each labeled label is one of multiple candidate class labels. The neural network model includes a first feature extraction network and a second feature extraction network. The feature extraction model is a first feature extraction network that meets the training termination condition.

[0011] During training, the label feature vector of each candidate category label is obtained through the second feature extraction network, the initial feature vector of each sample is obtained through the first feature extraction network, and the target feature vector of the sample is obtained by fusing the initial feature vector of the sample with the label feature vectors of each labeled label of the sample. The total training loss of the neural network model is determined based on the labeled labels and target feature vectors of each sample.

[0012] On the other hand, embodiments of this application also provide a data processing apparatus, which includes:

[0013] A data acquisition module is used to acquire data to be processed, wherein the data to be processed is data containing information of at least one modality in multimedia information;

[0014] The data processing module is used to extract feature vectors from the data to be processed using a trained feature extraction model, and to determine the category of the data to be processed based on the feature vectors, or to determine target data that matches the data to be processed from the dataset.

[0015] The feature extraction model is obtained by repeatedly training a neural network model with multiple samples having at least one labeled label. The labeled label is the true class label of the sample, and each labeled label is one of multiple candidate class labels. The neural network model includes a first feature extraction network and a second feature extraction network. The feature extraction model is the first feature extraction network that meets the training termination condition. During training, the label feature vector of each candidate class label is obtained through the second feature extraction network, and the initial feature vector of each sample is obtained through the first feature extraction network. The target feature vector of the sample is obtained by fusing the initial feature vector of the sample with the label feature vectors of each labeled label of the sample. The total training loss of the neural network model is determined based on the labeled labels and target feature vectors of each sample.

[0016] Optionally, the training process further includes: obtaining an initial semantic vector for each candidate category label; determining the number of co-occurrences between pairs of labels in the plurality of candidate category labels based on the labeled labels of the plurality of samples; for each candidate category label, determining the correlation between the label and each label in the plurality of candidate category labels based on the number of co-occurrences between the label and each label in the plurality of candidate category labels, wherein the correlation between two labels characterizes the co-occurrence probability between the two labels;

[0017] The label feature vector of each candidate category label is obtained in the following way:

[0018] Based on the initial semantic vector of each candidate category label and the correlation between each candidate category label and each of the plurality of candidate category labels, the label feature vector of each candidate category label is extracted by the second feature extraction network.

[0019] Optionally, the label feature vector for each candidate category label is obtained in the following way:

[0020] For each candidate category label, the non-relevance between the label and each of the plurality of candidate category labels is determined based on the correlation between the label and each of the plurality of candidate category labels;

[0021] The second feature extraction network performs at least one feature update operation, and based on the second feature vector of each candidate category label obtained from the last feature update operation, a label feature vector for each candidate category label is obtained.

[0022] The feature update operation includes the following steps:

[0023] Based on the semantic correlation between the initial semantic vectors of each candidate category label, a first feature vector corresponding to each candidate category label is obtained by performing feature fusion on the initial semantic vectors of each candidate category label.

[0024] For each candidate category label, the irrelevance between the label and each of the multiple candidate category labels is used as a weight to perform a weighted fusion of the first feature vectors of each of the multiple candidate category labels. The second feature vector of the label is obtained based on the weighted fused feature vector, and the second feature vector is used as the initial semantic vector of the label in the next feature update operation.

[0025] Optionally, the relevance between each of the candidate category labels and each of the plurality of candidate category labels is determined by the following method:

[0026] Based on the labeled tags of the multiple samples, determine the number of times each candidate category tag appears in the multiple samples;

[0027] For each candidate category label, the ratio of the number of times the label co-occurs with each of the multiple candidate category labels to the number of times the label appears is used as the co-occurrence probability between the label and each of the multiple candidate category labels. The correlation between the label and each of the multiple candidate category labels is obtained based on the co-occurrence probability.

[0028] Optionally, for each sample, the target feature vector of that sample is obtained in the following way:

[0029] If the sample has only one label, then the label feature vector of that label is used as the weight vector of the initial feature vector of the sample. The initial feature vector of the sample is weighted using this weight vector. Based on the weighted feature vector, the target feature vector of the sample is obtained.

[0030] If the sample has multiple labels, the label feature vectors of the multiple labels of the sample are fused together, and the fused label feature vector is used as the weight vector of the initial feature vector of the sample. The initial feature vector of the sample is weighted using the weight vector, and the target feature vector of the sample is obtained based on the weighted feature vector.

[0031] Optionally, for each sample, the target feature vector of that sample is obtained in the following way:

[0032] For each sample, the first feature extraction network performs the following operations to obtain the target feature vector of the sample: extract the initial feature vector of the sample; fuse the initial feature vector of the sample and the label feature vectors of each labeled label of the sample, and perform feature extraction on the fused feature vector to obtain the target feature vector of the sample.

[0033] Optionally, for each of the samples, the training operation further includes:

[0034] The target feature vector of the sample is quantized to obtain the quantized feature vector corresponding to the sample.

[0035] The total training loss of the neural network model is determined in the following way:

[0036] Based on the labeled tags and target feature vectors of each sample, a first training loss is determined; based on the difference between the target feature vector and the quantized feature vector of each sample, a second training loss is determined; based on the first training loss and the second training loss, the total training loss of the neural network model is determined.

[0037] Optionally, the total training loss of the neural network model is determined in the following way:

[0038] For each sample pair among the plurality of samples, a first similarity is determined based on the number of identical labels shared by the two samples in the sample pair, wherein each sample pair consists of two samples among the plurality of samples;

[0039] Based on the first similarity of each sample pair, a first sample pair and a second sample pair are determined in each sample pair, wherein the first similarity of the first sample pair is greater than the first similarity of the second sample pair;

[0040] For each of the first sample pairs, the training loss corresponding to the first sample pair is obtained based on the difference between the target feature vectors of the two samples in the first sample pair;

[0041] For each second sample pair, the training loss corresponding to the second sample pair is determined based on the second similarity between the target feature vectors of the two samples in the second sample pair;

[0042] The total training loss of the neural network model is obtained based on the training loss corresponding to each of the first sample pairs and the training loss corresponding to each of the second sample pairs.

[0043] Optionally, for each first sample pair, obtaining the training loss corresponding to the first sample pair based on the difference between the target feature vectors of the two samples in the first sample pair includes:

[0044] Determine the degree of difference between the target feature vectors of the two samples in the first sample pair;

[0045] Based on the first similarity corresponding to the first sample pair, the loss weight corresponding to the first sample pair is determined, wherein the loss weight is not less than 1, and the loss weight corresponding to each first sample pair is positively correlated with the first similarity corresponding to the first sample pair.

[0046] The first similarity of the first sample pair is used as the weight, and the difference of the first sample pair is weighted to obtain the training loss of the sample pair.

[0047] On the other hand, embodiments of this application provide a method for training a feature extraction model, the method comprising:

[0048] Obtain a training set, which includes multiple samples, each sample having at least one labeled label, the labeled label being the true class label of the sample, each labeled label being one of multiple candidate class labels, the sample being data containing information of at least one modality in multimedia information;

[0049] The training operation is repeatedly performed on the neural network model to be trained based on the training set until a preset condition is met. The neural network model includes a first feature extraction network and a second feature extraction network. The first feature extraction network that meets the preset condition is taken as the trained feature extraction model. The training operation includes the following steps:

[0050] The label feature vector of each candidate category label is obtained through the second feature extraction network;

[0051] For each sample, the initial feature vector of the sample is obtained through the first feature extraction network, and the target feature vector of the sample is obtained by fusing the initial feature vector with the label feature vectors of each labeled label of the sample.

[0052] Based on the labeled tags and target feature vectors of each sample, the total training loss of the neural network model is determined;

[0053] If the preset conditions are not met, the model parameters of the neural network model are adjusted based on the total training loss.

[0054] Optionally, at least some samples in the training set have multiple labeled tags.

[0055] Optionally, the method further includes: obtaining an initial semantic vector for each candidate category label, wherein obtaining the label feature vector for each candidate category label through the second feature extraction network includes:

[0056] Based on the initial semantic vectors of each candidate category label, the semantic correlation between each candidate category label is learned through the second feature extraction model. Based on the semantic correlation between each candidate category label and the initial semantic vectors of each candidate category label, the label feature vector of each candidate category label is obtained through feature fusion.

[0057] On the other hand, embodiments of this application provide a training apparatus for a feature extraction model, the apparatus comprising:

[0058] The training data acquisition module is used to acquire a training set, which includes multiple samples, each sample having at least one label, the label being the true class label of the sample, each label being one of multiple candidate class labels, and the sample being data containing information of at least one modality in multimedia information;

[0059] The model training module is used to repeatedly perform training operations on the neural network model to be trained based on the training set until a preset condition is met. The neural network model includes a first feature extraction network and a second feature extraction network. The first feature extraction network that meets the preset condition is used as the trained feature extraction model. The training operation includes the following steps:

[0060] The label feature vector of each candidate category label is obtained through the second feature extraction network;

[0061] For each sample, the initial feature vector of the sample is obtained through the first feature extraction network, and the target feature vector of the sample is obtained by fusing the initial feature vector with the label feature vectors of each labeled label of the sample.

[0062] Based on the labeled tags and target feature vectors of each sample, the total training loss of the neural network model is determined;

[0063] If the preset conditions are not met, the model parameters of the neural network model are adjusted based on the total training loss.

[0064] In another aspect, embodiments of this application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method provided in any optional embodiment of this application.

[0065] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in any optional embodiment of this application.

[0066] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the method provided in any optional embodiment of this application.

[0067] The beneficial effects of the technical solution provided in this application are as follows:

[0068] The data processing method provided in this application embodiment, when determining the category of the data to be processed or identifying target data matching the data to be processed from the dataset, can call a trained feature extraction model to extract the feature vector of the data to be processed, and determine the data category or target data based on the feature vector. In the process of training the neural network model based on the training set (i.e., the aforementioned multiple samples), the target feature vector of the sample includes not only the initial feature vector containing the semantic information of the sample itself extracted by the first feature extraction network, but also the label feature vector of each labeled label of the sample learned by the second feature extraction network. Since the label of each sample represents the true category of that sample, the target feature vector of each sample learned by the training method provided in this application contains information about the sample itself and the category representation information of the sample. The training method that constrains the parameter update of the model based on the total training loss obtained from the labels of each sample and the target feature vector allows the trained feature extraction model to better learn the semantic information of the data itself and the relevant category information of the data input into the model. This makes the feature vector of the data extracted by the model have better feature expression ability and discriminative power. Therefore, based on the feature vector of the data to be processed extracted by the feature extraction model, the category of the data to be processed can be determined more accurately, or the target data matching the data can be found more accurately. The solution provided in this application can better meet the needs of practical applications. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0070] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0071] Figure 2 A flowchart illustrating a training method for a feature extraction model provided in an embodiment of this application;

[0072] Figure 3 This is a schematic diagram of the structure of an image retrieval system provided in an embodiment of this application;

[0073] Figure 4 This application provides a schematic diagram of the image processing principle in an image retrieval system.

[0074] Figure 5 This is a schematic diagram illustrating the structure and data processing principle of a neural network model provided in an embodiment of this application;

[0075] Figure 6A schematic diagram illustrating the implementation process of the model training and application stages provided in this application embodiment;

[0076] Figure 7 and Figure 8 This is a schematic diagram of the structure of a second feature extraction network provided in the embodiments of this application;

[0077] Figure 9 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0078] Figure 10 This is a schematic diagram of the structure of an electronic device to which this application applies. Detailed Implementation

[0079] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0080] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.” When describing multiple (two or more) items, if the relationship between the multiple items is not explicitly defined, the multiple items can refer to one, several or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" can be implemented as parameter A includes A1 or A2 or A3, or it can be implemented as parameter A includes at least two of the three items A1, A2 and A3.

[0081] The solutions provided in this application relate to Artificial Intelligence (AI) technology, specifically the training and application of neural network models. For example, based on the training method provided in this application, machine learning (ML) can be used to train the neural network model, resulting in a feature extraction model with better performance. After training a feature extraction model that meets the application requirements, features can be extracted from the data to be processed based on this model. Based on the extracted features, the classification results of the data can be obtained, or the target data matching the data to be processed can be determined from the database based on the extracted features.

[0082] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. With the research and advancement of AI technology, it has already been researched and applied in many fields, including smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, smart customer service, vehicle networking, and intelligent transportation. It is believed that with further technological development, this technology will be applied in even more fields and play an increasingly important role.

[0083] Optionally, the data processing involved in the method provided in this application embodiment can be implemented based on cloud technology. For example, the training method provided in this application can be implemented based on cloud technology, and various data calculations involved in the training process can be performed using cloud computing. Optionally, the storage of each training set used in the training process can also be performed using cloud storage.

[0084] To better understand and explain the solutions provided in the embodiments of this application, the relevant technical terms involved in the embodiments of this application will be explained below.

[0085] Image recognition: This can be category-level recognition, which does not consider the specific instance of an object, but only the category of the object (such as dog, cat, bird, etc.) and identifies the category to which the object belongs. A typical example is the recognition task in the large-scale general object recognition open-source dataset ImageNet, which identifies which of 1000 categories a given object belongs to.

[0086] Binary quantization: For a D-dimensional feature vector embedding, the normalized vector typically takes values ​​ranging from -1 to 1 floating-point numbers. Compressing the features to a specified number of bits (e.g., 48 bits) to produce binary codes with values ​​of 0 and 1 (called 48-bit compression) is called vector binary quantization. The result of binary quantization can be called a binary code or a binary vector.

[0087] Binary quantization index: A binary vector with a finite number of bits is obtained from a D-dimensional feature vector through a certain calculation process (model). During retrieval, the binary vector is used as an index to recall images.

[0088] ImageNet: A large-scale, general-purpose open-source dataset for object recognition.

[0089] ImageNet pre-trained model: A deep learning network model trained based on ImageNet is obtained by calculating the parameter weights of the model.

[0090] Hamming distance: Used to measure the distance between binary features. It is achieved by counting the number of feature bits with different values. For example, the Hamming distance between binary vector 1000 and binary vector 0011 is 3.

[0091] Image semantic retrieval: The process of finding semantically related images from an image library for a query image. Semantically related images can be those with the same category (e.g., belonging to the dog category) or the same attributes (attributes can be multiple, such as yellow, happy, natural environment, etc.).

[0092] Label relevance: An image can have many labels, such as park, lake, and grassland. These labels can appear on the same image at the same time. It is often believed that two labels with a high probability of co-occurrence are related. For example, if park and grassland often appear together, the correlation is strong. However, if park and desert rarely appear together in the same image, the correlation is weak.

[0093] Currently, data retrieval and classification techniques based on deep learning-based data representations (such as image hash features) are gradually being applied in industry. Taking image hash features as an example, existing conventional methods for learning semantic hash features often ignore the correlation between semantic labels, which can easily lead to low discriminative power of hash features for co-occurring labels, causing the following problems:

[0094] 1) Two co-occurring but unrelated labels cause representation confusion due to co-occurrence. For example, information of one label is learned but mistakenly identified as information of another co-occurring label. For example, after learning the grass park label with a high co-occurrence probability, the grass image is also labeled as a park.

[0095] 2) Two co-occurring related tags, such as football field and grass, exist together and have many commonalities, often resulting in missed recalls between them;

[0096] 3) For differentiated samples, such as sample pairs with only one identical label and other dissimilar labels, hash features cannot effectively distinguish them. That is, samples with the same label may be characterized as similar, but there is no difference in the degree of similarity. However, in practical applications, we hope that the hash feature similarity between sample pairs with two identical labels is greater than that between sample pairs with one identical label.

[0097] Because current neural network-based feature representations have low discriminative power for co-occurrence labels, they often result in numerous false positives in retrieval and other applications. To address at least one or more of these problems and better meet application requirements, this application provides a novel method for training a feature extraction model.

[0098] The training method or data processing method of the feature extraction model provided in this application embodiment can be executed by any computer device, such as a terminal device or a server. For example, the trained feature extraction model can be deployed to the server of an application. Users can send the data to be processed to the server through user terminals. The server can call the feature extraction model to extract features from the data to be processed and obtain the processing result of the data to be processed based on the extracted feature vectors. For example, the feature extraction model can be used as the feature extraction module of a classification model. Based on the extracted feature vectors, the category of the data to be processed can be determined. Alternatively, the data to be processed can be an image in a search request. Based on the feature vector of the image, a target image matching the image can be found in the dataset, such as the image with the highest similarity to the image in the database, or an image in the database with a similarity exceeding a set threshold.

[0099] The aforementioned server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The aforementioned user terminal (also referred to as user equipment) can be a smartphone, tablet computer, laptop computer, desktop computer, intelligent voice interaction device (e.g., smart speaker), wearable electronic device (e.g., smartwatch), in-vehicle terminal, smart home appliance (e.g., smart TV), AR / VR device, etc., but is not limited thereto. The terminal and server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions.

[0100] Optionally, the data processing method provided in this application embodiment can be implemented as an independent application or a functional module / plugin of an application. The server of the application can process the data to be processed by calling a trained feature extraction model. For example, the application can be an application with image retrieval function. The server of the application can retrieve similar images from the image database according to the image retrieval request, or it can perform deduplication processing on the images in the database based on the feature vectors of each image in the database.

[0101] The data processing method provided in this application can be applied to any application scenario that requires extracting feature vectors from the data to be processed, including but not limited to the above-mentioned scenarios such as data classification, data query, or data deduplication.

[0102] The technical solutions of this application and their effects are described below through several exemplary embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0103] Figure 1 This application provides a schematic flowchart of a data processing method according to an embodiment of the present application. Figure 1 As shown, the method may include:

[0104] Step S110: Obtain data to be processed, wherein the data to be processed is data containing information of at least one modality in multimedia information;

[0105] Step S120: Extract the feature vector of the data to be processed using the trained feature extraction model;

[0106] Step S130: Based on the obtained feature vector, determine the category of the data to be processed, or identify the target data that matches the data to be processed from the dataset.

[0107] The aforementioned feature extraction model is obtained by repeatedly training a neural network model with multiple samples having at least one labeled label. The labeled label of the sample is the true class label of the sample, and each labeled label is one of multiple candidate class labels. The neural network model includes a first feature extraction network and a second feature extraction network. The feature extraction model is the first feature extraction network that meets the training termination condition. During the training process, the label feature vector of each candidate class label is obtained through the second feature extraction network, and the initial feature vector of each sample is obtained through the first feature extraction network. The target feature vector of the sample is obtained by fusing the initial feature vector of the sample with the label feature vectors of each labeled label of the sample. The total training loss of the neural network model is determined based on the labeled labels and target feature vectors of each sample.

[0108] In this application embodiment, the data format of the data to be processed is not limited. The data to be processed can be data containing information of at least one modality among multimedia information. Modality refers to the source or form of information, which can be the medium of information. The data to be processed can include one or more modalities such as images, text, or audio. Similarly, the data format of the samples in the training set (i.e., the multiple samples mentioned above) corresponds to the data format of the data to be processed. For different application scenarios, the modality of the samples in the training set can be different. The samples can be data of images, audio, text, video, or other modalities.

[0109] In this embodiment of the application, each sample has one or more labels. Optionally, at least some samples in the training set have at least two labels. The label of a sample represents the true category of the sample, i.e., the true category label. Optionally, the label can be a category name. Multiple candidate category labels can be understood as a label set. The labels of all samples in the training set are labels in this label set. That is, the candidate category labels in the label set include all the labels of all samples in the training set.

[0110] Similarly, for the data to be processed, there can be at least one category. Optionally, based on the extracted feature vector, the probability that the data to be processed belongs to each candidate category (i.e., candidate category label) can be predicted. The candidate category corresponding to the probability greater than a set threshold is the category of the data to be processed.

[0111] For the data query applications mentioned above, the dataset (or database) can be a retrieval database, the target data can be the retrieval results, the dataset can also be a dataset that needs to be deduplicated, and the target data can be potential duplicate or similar data of the data to be processed found in the query. Optionally, the feature vector of each data in the dataset can be extracted by a feature extraction model, and the target data can be determined by calculating the similarity between the feature vector of the data to be processed and the feature vector of each data in the dataset. For example, the data in the dataset with a similarity greater than a set threshold with the data to be processed can be determined as the target data, or the data in the dataset can be sorted in descending order of similarity, and the data at the top of the sorted list can be determined as the target data.

[0112] Based on the method provided in the embodiments of this application, the feature vector of the obtained data can have better feature expression ability and discriminability. Therefore, it can effectively improve the data processing effect, that is, it can more accurately determine the category of the data to be processed, or more accurately find the target data that matches the data, and better meet the needs of practical applications.

[0113] The training method of the feature extraction model provided in this application will be described below with reference to a variety of optional embodiments.

[0114] Figure 2 This document illustrates a flowchart of a training method for the aforementioned feature extraction model provided in an embodiment of this application. This method can be executed by any electronic device; optionally, it can be executed by a server (e.g., a cloud server). The following description uses a training server as the execution entity. Figure 1 As shown, the method includes steps S210 to S240.

[0115] Step S210: Obtain a training set, which includes multiple samples, each sample having at least one labeled label, the labeled label being the true class label of the sample, and each labeled label being one of multiple candidate class labels.

[0116] The embodiments of this application do not limit the method of obtaining the training set or the specific classification method of the samples in the training set. For example, the training set can be configured according to the specific application scenario. For example, it can be collected through the network under the premise of complying with relevant regulations. The labels of the samples can be manually labeled according to the preset labeling rules. Alternatively, the training set can be an open source training dataset, such as the training dataset of an existing multi-label classification model.

[0117] As an example, the training set can be an image dataset, and the samples are training images (i.e., sample images). One or more labels on the training images can be manually labeled, or at least one label on the training images can be predicted using other existing image multi-label classification models, and the predicted label can be used as the label of the training images.

[0118] Step S220: Repeatedly perform training operations on the neural network model to be trained based on the training set until the preset conditions, i.e. the training termination conditions, are met. The neural network model includes a first feature extraction network and a second feature extraction network. The first feature extraction network that meets the preset conditions is taken as the trained feature extraction model.

[0119] The training operations mentioned above may include, for example: Figure 2 Steps S221 to S224 shown:

[0120] Step S221: Obtain the label feature vector for each candidate category label through the second feature extraction network;

[0121] Step S222: For each sample, the initial feature vector of the sample is obtained through the first feature extraction network, and the target feature vector of the sample is obtained by fusing the initial feature vector with the label feature vectors of each labeled label of the sample.

[0122] Step S223: Determine the total training loss of the neural network model based on the labeled tags and target feature vectors of each sample;

[0123] Step S224: If the preset conditions are not met, adjust the model parameters of the neural network model based on the total training loss.

[0124] The training termination condition of the neural network model can be configured according to the requirements. Optional preset conditions may include the number of training cycles (i.e., training epochs, a complete training set is passed through the neural network once and returned once, this process is called one epoch) reaching a set number, or the total training loss of the model meeting certain conditions, which is usually referred to as the convergence of the model's loss function, such as the first training loss being less than a set value or the training loss corresponding to a set number of consecutive training operations being less than the set value.

[0125] In this embodiment, the first feature extraction network is the initial model structure of the feature extraction model to be trained, and the second feature extraction network is an auxiliary training network for the first feature extraction network, used to improve the model performance of the trained first feature extraction network. When training the neural network model based on the training set, the first feature extraction network and the second feature extraction network can be iteratively trained simultaneously (i.e., model parameter learning) until the first feature extraction network that meets the above-mentioned preset conditions is obtained.

[0126] As can be seen from the above training process, the first feature extraction network is used to extract the feature vectors of the samples, and the second feature extraction network is used to obtain the label feature vectors of the candidate category labels. The specific model structures of the first and second feature extraction networks are not uniquely limited in this embodiment. Optionally, the first feature extraction network may include, but is not limited to, a deep feature extraction model based on a convolutional neural network (CNN). This network may include at least one feature extraction layer (optionally, the feature extraction layer may include convolution calculations, ReLU nonlinear activation function calculations, pooling calculations, etc.) and at least one feature mapping layer. For example, the feature extraction layer may include a convolutional layer, and the feature mapping layer may be a fully connected layer. When there are multiple fully connected layers, activation function layers may also connect adjacent fully connected layers. For each sample, features can be extracted through the feature extraction layer of the first feature extraction network. The features extracted by the feature extraction layer generate the initial feature vector of the sample through the fully connected layer. Then, the target feature vector of the sample can be obtained by fusing the initial feature vector with the label feature vectors of each labeled label of the sample extracted by the second feature extraction network.

[0127] Optionally, for each sample, the initial features of the sample and the label feature vectors of each labeled label of the sample can be fused to obtain a feature vector, which can be used as the target feature vector of the sample. Alternatively, the first feature extraction network can perform the following operations on the sample to obtain the target feature vector:

[0128] Extract the initial feature vector of the sample;

[0129] The initial feature vector of the sample and the label feature vectors of each labeled label of the sample are fused together, and feature extraction is performed on the fused feature vector to obtain the target feature vector of the sample.

[0130] Regarding the specific method for fusing the initial feature vector of each sample with the label feature vectors of each labeled label, this application embodiment does not impose a unique limitation. Optionally, the initial feature vector and the label feature vectors of each labeled label can be vector-added, that is, the element values ​​at corresponding positions of the vectors are added together, and the resulting vector is used as the target feature vector. Alternatively, the first feature extraction network can perform feature extraction on the added vector to obtain the target feature vector. Optionally, for each sample, the initial feature vector of the sample and the label feature vector of the sample's labeled label can be vector-multiplied to obtain a weighted feature vector, and the weighted vector is used as the target feature vector. Alternatively, the first feature extraction network can perform feature extraction on the weighted vector to obtain the target feature vector. When obtaining the target feature vector using the multiplication method, the feature vectors of the sample's labeled label can be regarded as the weights of the initial feature vector. Optionally, the element values ​​in the label feature vector can be numbers ranging from 0 to 1.

[0131] Optionally, for each sample, the target feature vector of the sample is obtained by fusing the initial feature vector with the label feature vectors of each labeled label of the sample, including:

[0132] If the sample has only one label, then the label feature vector of that label is used as the weight vector of the initial feature vector of the sample. The initial feature vector of the sample is weighted using this weight vector. Based on the weighted feature vector, the target feature vector of the sample is obtained.

[0133] If the sample has multiple labels, the label feature vectors of the multiple labels of the sample are fused together, and the fused label feature vector is used as the weight vector of the initial feature vector of the sample. The initial feature vector of the sample is weighted using the weight vector, and the target feature vector of the sample is obtained based on the weighted feature vector.

[0134] In other words, if a sample has multiple labels, the label feature vectors of multiple labels can be fused first (e.g., by calculating the mean), and then the initial feature vector can be calculated by weighted multiplication using the fused feature vector. The weighted feature vector can then be used as the target feature vector of the sample. Alternatively, the weighted feature vector can be used to extract features again to obtain the target feature vector of the sample.

[0135] It is understandable that after obtaining the trained first feature extraction network, when using this network to obtain the feature vector of the data to be processed, the aforementioned feature fusion step does not exist, because the trained feature extraction model does not include the structure of the second feature extraction network when it is used.

[0136] The second feature extraction network can also include at least one layer of feature extraction layers and at least one layer of feature mapping layers. This network is used to extract feature vectors for each candidate category label. Each candidate category label can be the category name, such as the Chinese name of the category. Based on the Chinese name corresponding to each candidate category label, the second feature extraction network can obtain the feature vector (i.e., the label feature vector) corresponding to that name. The names of each label can be directly input into the second feature extraction network, which will obtain the corresponding feature vector through feature embedding, feature extraction, and feature mapping. Alternatively, an initial feature vector for each candidate label can be generated first using other pre-trained feature extraction models, and then the second feature extraction network can obtain the label feature vector for each candidate category label based on this initial feature vector.

[0137] Understandably, in practice, the training set can be divided into multiple batches of training data, each batch containing multiple samples. Each training operation can use samples from one batch. For each sample in a batch, after obtaining the target feature vector of each sample through the neural network model, if the preset conditions are not met, the total training loss of the model can be calculated based on the labeled data and target feature vector of each sample. The model parameters of the neural network model can then be adjusted based on the total training loss. For example, the model parameters of the first and second feature extraction networks can be adjusted using the gradient descent algorithm. The above training operation is then performed on the neural network model with adjusted parameters based on the training set.

[0138] If the total training loss of the model meets certain conditions or the number of training iterations reaches a set number, the model can be considered to have met the training termination condition. The first feature extraction network at this point can be used as the trained feature extraction network. Of course, after obtaining the first feature extraction network that meets the conditions, the network can also be tested for performance, such as testing the network's performance based on samples in the test set. If the test passes, the network at this point can be used as the trained feature extraction model. If the test fails, the above neural network model can continue to be trained.

[0139] The specific form of the total training loss of the model is not limited in the embodiments of this application. Optionally, for each training operation, after obtaining the target feature vector of each sample, the total training loss of the model can be calculated based on the difference between the target feature vector and the label of each sample. For example, the above neural network model may also include a classification model, which is used to predict the category of each sample based on the target feature vector of each sample. The total training loss can be calculated based on the predicted category and the true category label of each sample. This loss represents the difference between the predicted category and the true category of each sample.

[0140] As another alternative, the method of determining the total training loss of the neural network model based on the labeled tags and target feature vectors of each sample can include:

[0141] For each sample pair among the above multiple samples, the first similarity of the sample pair is determined based on the number of identical labeled tags that the two samples in the sample pair have;

[0142] Based on the first similarity of each sample pair, determine the first sample pair and the second sample pair in each sample pair, wherein the first similarity of the first sample pair is greater than the first similarity of the second sample pair;

[0143] For each first sample pair, the training loss corresponding to the first sample pair is obtained based on the difference between the target feature vectors of the two samples in the first sample pair;

[0144] For each second sample pair, the training loss corresponding to the second sample pair is determined based on the second similarity between the target feature vectors of the two samples in the second sample pair.

[0145] The total training loss of the neural network model is obtained based on the training loss corresponding to each first sample pair and the training loss corresponding to each second sample pair.

[0146] It is understandable that each sample pair mentioned above refers to any two samples in the training set. Assuming the number of samples in a batch used in one training operation is *bs*, then these *bs* samples generate *bs**(*bs*-1) / 2 sample pairs. Since each sample in the training set has a label, for each sample pair, the first similarity between the two samples in the pair can be determined based on their labels. The more common labels between the two samples, the higher the first similarity. After determining the first similarity between each sample pair, it is possible to identify which sample pairs are similar (i.e., the first sample pair, also known as positive sample pairs) and which are dissimilar (i.e., the second sample pair, also known as negative sample pairs) based on this similarity.

[0147] Optionally, for each sample pair, if the first similarity corresponding to the sample pair is greater than a set threshold, then the sample pair is determined as the first sample pair; if the first similarity corresponding to the sample pair is less than or equal to the set threshold, then the sample pair is determined as the second sample pair. The set threshold can be configured according to actual needs. The set threshold can be a positive number not less than 0. Sample pairs with a first similarity greater than the set threshold are considered similar sample pairs, and sample pairs with a first similarity less than or equal to the set threshold are considered dissimilar sample pairs.

[0148] Since the purpose of training a neural network model is to enable the first feature extraction network to learn target feature vectors that can better represent samples, and compared to dissimilar sample pairs, the two samples in a similar sample pair should have more similar feature vectors, after determining each first sample pair and second sample pair, the total training loss of the model can be calculated based on the difference between the target feature vectors of the two samples in each first sample pair (e.g., 1 minus the similarity) and the similarity between the target feature vectors of the two samples in each second sample pair. Thus, the training of the model is constrained based on the total training loss, so that the difference between the target feature vectors of similar sample pairs learned by the model becomes smaller and smaller, and the difference between the target feature vectors of dissimilar sample pairs becomes larger and larger.

[0149] Optionally, after obtaining the target feature vectors of each sample through the neural network model, for each sample pair, the second similarity between the target feature vectors of the two samples in the sample pair can be calculated. For the first sample pair, the second similarity corresponding to the sample pair is subtracted from 1 to obtain the difference corresponding to the sample pair, and the difference is used as the training loss corresponding to the sample pair. For the second sample pair, the second similarity between the sample pairs can be used directly as the training loss corresponding to the sample pair. After obtaining the training loss corresponding to each sample pair, operations such as summing or averaging the training losses corresponding to these sample pairs can be performed, and the total training loss of the neural network model is obtained based on the operation results.

[0150] The training method provided in this application, when acquiring the target feature vector of a sample, in addition to learning an initial feature vector containing the semantic information of the sample through a first feature extraction network, also learns the label feature vector of each candidate category label through a second feature network in order to better learn the difference and correlation information between different samples. For each sample, the target feature vector of the sample is obtained by fusing the initial feature vector of the sample and the label feature vectors of each labeled label of the sample. Since the labeled label of each sample represents the true category of the sample, the target feature vector of each sample learned by the method provided in this application incorporates the semantic information of the sample itself and the semantic information of one or more true categories to which the sample belongs. The feature extraction model trained based on the labeled labels of each sample and the above-mentioned target feature vector can better learn the semantic information of the data itself and the category information of the data input into the model, and obtain a more discriminative feature vector. The feature extraction model trained in this way can better meet the needs of practical applications, especially in multi-label classification application scenarios. For the data to be processed, the model can obtain a feature vector with better feature expression ability, which can effectively characterize the processing effect of the data.

[0151] In some practical applications, to improve processing efficiency, the feature vectors of data are quantized. This transforms the image's feature vectors into feature vectors with a specified dimension and elements taking specific values. For example, in image retrieval, after obtaining the image's feature vectors, binary quantization can be performed to convert them into binary codes of 0s and 1s, resulting in a binary feature vector. Using this binary vector for image retrieval can significantly improve retrieval efficiency.

[0152] To better meet the needs of practical applications, in an optional embodiment of this application, the above training operation may further include:

[0153] For each sample, the target feature vector of the sample is quantized to obtain the quantized feature vector corresponding to the sample.

[0154] The above-mentioned total training loss of the neural network model, determined based on the labeled tags and target feature vectors of each sample, includes:

[0155] The first training loss is determined based on the labeled tags and target feature vectors of each sample;

[0156] The second training loss is determined based on the difference between the target feature vector and the quantized feature vector of each sample.

[0157] The total training loss of the neural network model is determined based on the first training loss and the second training loss.

[0158] Optionally, in this alternative approach, the total training loss of the model includes two parts: a similarity loss, i.e., the first training loss, and a quantization loss, i.e., the second training loss. The two parts of the loss can be added together to obtain the total training loss of the model, or the two parts of the loss can be pre-configured with their respective weights and the two parts of the loss can be weighted and summed to obtain the total training loss.

[0159] In practical applications, if quantized feature vectors are required, after obtaining the feature vectors of the data to be processed through a trained feature extraction model, these feature vectors need to be mapped to specific values. If the difference between the feature vectors before and after quantization is too large, even if the model can learn feature vectors with good feature representation capabilities, the data processing effect based on the quantized feature vectors cannot be guaranteed. To avoid this problem, in the alternative solution of this application, for application scenarios requiring feature quantization, in addition to calculating the similarity loss based on the labeled tags and target feature vectors of each sample during model training, the aforementioned quantization loss is further considered to ensure that the feature vectors before and after quantization have sufficiently good feature representation capabilities.

[0160] The specific method for quantizing the target feature vector of the sample can be configured according to actual application requirements, and this application embodiment does not limit it. Optionally, the target feature vector can be quantized using a pre-selected quantization function (such as the sign function), and the second training loss can be obtained by calculating the difference between the feature vectors of each sample before and after quantization. The training loss of the model can be obtained by summing the first training loss and the second training loss. Optionally, the weights corresponding to the first training loss and the second training loss can be obtained, and the two parts of the loss can be weighted and summed to obtain the total training loss of the model.

[0161] Optionally, in this embodiment, the target feature vector of the sample can be the sample's hash feature. After obtaining the sample's hash feature through a neural network model, the hash feature can be quantized (e.g., binary quantization) to obtain the quantized feature vector. In practical applications, such as data retrieval scenarios, data retrieval based on quantized feature vectors can effectively reduce the amount of data computation.

[0162] To further improve the discriminative power of the target feature vectors of the samples learned by the model, in an optional embodiment of this application, for each first sample pair, the training loss corresponding to the first sample pair is obtained based on the difference between the target feature vectors of the two samples in the first sample pair, including:

[0163] Determine the degree of difference between the target feature vectors of the two samples in the first sample pair;

[0164] Based on the first similarity of the first sample pair, the loss weight of the first sample pair is determined, wherein the loss weight is not less than 1, and the loss weight of each first sample pair is positively correlated with the first similarity of the first sample pair.

[0165] The first similarity of the first sample pair is used as the weight, and the difference of the first sample pair is weighted to obtain the training loss of the sample pair.

[0166] For any pair of samples, the first similarity between the two samples is determined by the labels of the two samples in the pair. The more labels that the two samples have in common, the greater the first similarity between the two samples. Since the first similarity corresponding to different similarity sample pairs may be different, in order to ensure that the similarity between the target feature vectors of two samples in a similar sample pair with a relatively higher first similarity learned by the neural network model is greater than the similarity between similar sample pairs with a relatively lower first similarity, the scheme provided in this application further considers the first similarity between the first sample pairs when calculating the training loss corresponding to each first sample pair. Specifically, based on the first similarity corresponding to the sample pair, the loss weight (weight of difference) corresponding to the first sample pair is determined, and the weight is not less than 1. The difference between the first sample pairs is weighted using this weight to obtain the loss weight corresponding to the first sample pair. That is, a larger weight is assigned to the loss (difference) corresponding to the similar sample pair with a relatively higher first similarity. In order to make the total training loss of the model satisfy the learning strategy of tending to 0 or being as small as possible, during the training process of the model, for two samples in a similar sample pair with a relatively higher first similarity, the difference between the target feature vectors of the two samples that the model needs to learn is relatively smaller. That is, the similarity between the target feature vectors of two samples with a higher first similarity is higher.

[0167] The specific method for determining the loss weight corresponding to each first sample pair is not limited in this embodiment. Theoretically, it is sufficient to satisfy the requirement that the more identical labels a sample pair has, the greater its corresponding loss weight. Optionally, the first similarity between two samples can be equal to the number of identical labels the two samples have. The threshold value for determining whether a sample pair is a first sample pair or a second sample pair can be 0. That is, as long as two samples have at least one identical label, they are considered to be a first sample pair, and the first similarity between the sample pairs is the number of identical labels the two samples in the sample pair have. The first similarity corresponding to the first sample pair can be used as the loss weight corresponding to the sample pair. For example, if two samples have one identical label, the first similarity between the two samples and the corresponding loss weight are both 1.

[0168] During the training of the model, the learning of the label feature vector for each candidate category label can be done by directly learning it through the second feature extraction network based on the label name of the category label, or by first obtaining the initial semantic vector of each candidate category label, and then learning the label feature vector that can better distinguish different labels through the second feature extraction network based on the initial semantic vector of each candidate category label.

[0169] In this embodiment of the application, at least some samples in the training set may include at least two labels, that is, a sample may have multiple labels at the same time. For example, for an image containing multiple different objects, the label of the image may include labels corresponding to multiple different objects at the same time. For example, if the image includes a park, a lake and a grassland, the image may include three category labels corresponding to the park, the lake and the grassland respectively. The labels between two different samples may be completely different, partially the same or completely the same.

[0170] In practical applications, some category labels often co-occur, meaning they have a high co-occurrence probability. Labels with a high co-occurrence probability are usually related. For example, "park" and "grassland" often appear together, indicating a strong correlation between these two labels. Conversely, "park" and "desert" rarely co-occur, suggesting a weak correlation between these two labels. Considering the differences in semantic relevance between different candidate category labels, the training method provided in this application, when integrating the label feature vectors of each labeled label of a sample into the target feature vector of the sample, also incorporates the relevance between different category labels into the model training. This further enhances the representational ability of the feature vectors (i.e., feature representations) of the data extracted by the trained feature extraction model. Specifically, the training method provided in this application may further include:

[0171] Obtain the initial semantic vector for each candidate category label;

[0172] Based on the labeled labels of multiple samples, determine the number of co-occurrences between pairs of labels in multiple candidate category labels;

[0173] For each candidate category label, the correlation between the label and each of the multiple candidate category labels is determined based on the number of times the label co-occurs with each of the multiple candidate category labels. The correlation between two labels represents the co-occurrence probability between the two labels.

[0174] The above-mentioned method of obtaining the label feature vector for each candidate category label through the second feature extraction network may include:

[0175] Based on the initial semantic vectors of each candidate category label and the correlation between each candidate category label and each label among multiple candidate category labels, the label feature vector of each candidate category label is extracted through the second feature extraction network.

[0176] Based on this alternative approach, when obtaining the label feature vector of each candidate category label through the second feature extraction network, the correlation between candidate category labels can be used as guiding information. This allows the second feature extraction network to learn more discriminative label feature vectors of different labels based on the initial semantic vectors of each candidate category label and the correlation between candidate category labels. As a result, when the initial feature vector of a sample is fused with the labeled label (one or more of the candidate category labels) to obtain the target feature vector of the sample, the discriminative ability of the target feature vectors of different samples is improved, thus enhancing the performance of the model.

[0177] The method for obtaining the initial semantic vector of each candidate category label is not limited in this embodiment. Optionally, the initial semantic vector of each candidate category label can be obtained through a pre-trained feature extraction model. For example, the label names of each candidate category label can be input into a feature extraction network based on the Fasttext model architecture, and the Fasttext model can be used to embed the features of the label names of each candidate category label to obtain the initial semantic vector of each candidate category label.

[0178] In the method provided in this application embodiment, for each candidate category label, by extracting dense features, an initial semantic vector containing the semantic information of the category label can be obtained. Compared with the method of obtaining the label vector of the sample by using one-hot encoding, based on the initial semantic vector of each category label, the second feature extraction network can learn the semantic representation of each category label better. Moreover, the addition of the correlation between each candidate category label allows the second feature extraction network to learn better according to the strength of the correlation between the category labels.

[0179] As an optional approach, for each candidate category label, the correlation between the label and any other label among the multiple candidate category labels can be positively correlated with the number of times the label and any other label co-occur. For example, the co-occurrence counts among all labels can be normalized to obtain the correlation between labels. To obtain the correlation between different labels more accurately, this application embodiment also provides another optional approach. Specifically, for each candidate category label, determining the correlation between the label and each label among the multiple candidate category labels based on the number of times the label co-occurs with each other among the multiple candidate category labels can include:

[0180] Based on the labeled labels of multiple samples, determine the number of times each candidate category label appears in multiple samples;

[0181] For each candidate category label, the ratio of the number of times the label co-occurs with each of the multiple candidate category labels to the number of times the label appears is used as the co-occurrence probability between the label and each of the multiple candidate category labels. Based on the co-occurrence probability between the label and each of the multiple candidate category labels, the correlation between the label and each of the multiple candidate category labels is obtained.

[0182] In this alternative approach, the correlation between a candidate category label A and another label B is determined by the number of times label A and label B co-occur, and the number of times label A appears in each sample in the training set. For any two labels, the number of times they co-occur refers to the number of samples whose labeled labels simultaneously contain both labels. For example, if there are N1 samples in the training set, and N2 of these N1 samples have label A, then label A appears N2 times. If N3 of these N2 samples have label B, then label A and label B co-occur together N3 times. The ratio of N3 to N2 can be used as the co-occurrence probability of label A and label B. A higher co-occurrence probability indicates a higher correlation between label A and label B. It is understood that in this alternative approach, the co-occurrence probability of label A and label B is not necessarily equal to the co-occurrence probability of label B and label A.

[0183] In this alternative approach, the frequency of occurrence of each candidate category label is further considered. The co-occurrence probability between the label and each of the candidate category labels is calculated as the ratio of the co-occurrence frequency of the label to the total frequency of the label. The correlation between labels is then determined based on this co-occurrence probability. This approach not only normalizes the co-occurrence probability by using the frequency of each label in the sample but also makes the determined correlation between labels more objective and accurate. This is because, assuming the co-occurrence frequency of two candidate category labels A and B in all samples is t, but label A occurs 50 times and label B occurs 100 times, then for label A, the co-occurrence probability between label A and label B is t / 50, and for label B, the co-occurrence probability between label B and label A is t / 100. Compared to simply using the frequency of co-occurrence to calculate the probability between two labels, this method provides a more objective co-occurrence probability relative to each label, rather than assuming that for two labels with the same frequency of co-occurrence, whether A is relative to B or B is relative to A, the co-occurrence probability is the same. .

[0184] Optionally, after determining the co-occurrence probability between tags, this probability can be directly used as the correlation between tags. Alternatively, the co-occurrence probability can be preprocessed to obtain the correlation between tags. For example, a probability threshold can be preset. If the co-occurrence probability between tags is less than the probability threshold, the correlation between tags can be set to 0. If the co-occurrence probability between tags is greater than or equal to the probability threshold, the co-occurrence probability between tags can be directly used as the correlation between tags. It is understandable that, assuming there are Nc candidate category tags, for each tag, Nc correlations can be calculated. A tag's correlation with itself is 1. By calculating the correlations between each tag and the Nc tags, a correlation matrix can be obtained. This matrix has Nc rows and Nc columns, and each element in each row or column represents the correlation between a tag and the Nc tags.

[0185] As an alternative approach, the label feature vector for each candidate category label can be obtained in the following way:

[0186] For each candidate category label, determine the non-relevance between the label and each of the multiple candidate category labels based on the relevance between the label and each of the multiple candidate category labels;

[0187] The second feature extraction network performs at least one feature update operation, using the second feature vector of each candidate category label obtained from the last feature update operation as the label feature vector of each candidate category label; wherein the above feature update operation includes the following steps:

[0188] Based on the semantic correlation between the initial semantic vectors of each candidate category label, the first feature vector corresponding to each candidate category label is obtained by feature fusion of the initial semantic vectors of each candidate category label.

[0189] For each candidate category label, the irrelevance between the label and each of the multiple candidate category labels is used as the weight. The first feature vectors of each of the multiple candidate category labels are weighted and fused. The second feature vector of the label is obtained based on the weighted and fused feature vector. The second feature vector is used as the initial semantic vector of the label in the next feature update operation.

[0190] Optionally, for each candidate category label, the semantic vectors of each label in the multiple candidate category labels can be fused based on the correlation between the semantic vector of this label and the semantic vectors of each label in the multiple candidate category labels to obtain the first feature vector of this label. Specifically, the correlation calculation can be based on the initial semantic vector of the label, or it can be based on the feature vector of the label after further feature extraction from the initial semantic vector of the label.

[0191] In this alternative approach, the second feature extraction network can be an attention-based feature extraction network in actual implementation. For example, the first feature extraction network can include one or more cascaded attention layers (each attention layer corresponds to a feature update operation). By adopting the attention mechanism, the second feature extraction network can learn the semantic correlation between the initial semantic vectors of each candidate category label and achieve the fusion of the semantic vectors of each candidate category label to obtain the first feature vector of each candidate category label. Optionally, the initial semantic vectors of each candidate category label can be used as an input vector sequence. A second feature extraction network based on a self-attention mechanism can be used to extract the first feature vector of each candidate category label. Specifically, the above input vector sequence can be used as the input of the second feature extraction model. The second feature extraction network extracts features from the initial semantic vector of each candidate category label in the sequence to obtain the query vector q (i.e., query vector), key vector k (i.e., vector key), and value vector v (i.e. vector value) corresponding to each candidate category label. For each candidate category label, the correlation between the label and each label can be calculated based on the q of the label and the k of each label among multiple candidate category labels, which is the attention score. The correlation corresponding to each label is used as a weight to perform a weighted sum of the v of each label among multiple candidate category labels to obtain the first feature vector corresponding to the label.

[0192] In this alternative scheme of the application, since each first feature vector integrates the semantic information of multiple candidate category labels, and the integration is based on the correlation between labels, that is, the first feature vector of each candidate category label contains the semantic information of each label that is related to that label, and the greater the correlation, the more semantic information of the corresponding category labels it contains. Therefore, the first feature vector of the label learned through this scheme can achieve the learning of the correlation between labels, and can effectively learn the common information of related labels. Thus, when the label feature vector of the labeled label is integrated into the target feature vector of the sample, the target feature of the sample with high label correlation can contain the common information between labels, thereby improving the representation ability of the target feature vector.

[0193] Furthermore, since the correlation between labels has already been learned in the first feature vector of each candidate category label obtained through the above processing, if too much attention is paid to the common information between labels while ignoring the characteristic information of each label itself, i.e. the differences between different labels, for labels that are highly correlated but different, it is very likely that the feature vectors of highly correlated labels will not be highly distinguishable, resulting in the label feature vectors of different labels being unable to be effectively distinguished. To address this issue, the proposed solution, after obtaining the first feature vector of each candidate category label, can process the first feature vector by using the correlation between labels determined based on the co-occurrence frequency among candidate category labels. Specifically, for each candidate category label, the non-correlation between this label and each label among multiple candidate category labels can be used to weight the first feature vectors of each corresponding label among multiple candidate category labels, and the weighted feature vectors of each label can be fused (e.g., summed) to obtain the second feature vector of the candidate category label. The second feature vector obtained in this way can remove the correlation between each candidate category label, learn a more discriminative second feature vector for each candidate category label, and further improve the discriminability between samples with co-occurring labels based on the label feature vectors obtained in this way.

[0194] The data processing method provided in this application embodiment can be applied to any data processing task that requires data processing based on the feature vector of the data. It can include computers and is not limited to the above-mentioned data classification or data retrieval (determining target data). It is understood that the above-mentioned data retrieval can include scenarios such as data search and data deduplication. For example, the data to be processed can be the retrieved data in the data retrieval request, and the dataset is the database being retrieved. For another example, the data to be processed can be the query data in the deduplication request, and the dataset is the database that needs to be deduplicated.

[0195] To better illustrate and understand the solutions provided in the embodiments of this application and their practical value, the solutions provided in the embodiments of this application will be described below in conjunction with specific application scenarios.

[0196] The solution provided in this application embodiment can be applied to image deduplication retrieval. Accordingly, in this scenario, the samples in the training set are sample images, the labels are the category labels of the sample images, and the data to be processed are the images to be processed. Figure 3 The diagram shows a schematic representation of an image retrieval system applicable to this scenario embodiment. Figure 3As shown, the system may include a training server 10, a retrieval server 20, and a user terminal 30. The training server 10 can be used to execute the training method provided in the embodiments of this application to obtain a trained feature extraction model. The trained feature extraction model can be deployed to the retrieval server 20. The retrieval server 20 is a server for an application that provides retrieval services. Users of the application can initiate a data retrieval request through the client of the application on their user terminals. The request includes a retrieval image (i.e., an image specified by the user). After receiving the retrieval request, the retrieval server 20 can extract the image representation vector, i.e., the feature vector, of the retrieval image by calling the feature extraction model, and find several images that match the retrieval image from the image database based on the feature vector, and display these several images as retrieval results to the user terminal.

[0197] Figure 4 This illustration shows a schematic diagram of the structure and data processing principle of a neural network model provided in an embodiment of this application. Figure 5 A schematic diagram illustrating the principle of image retrieval based on a trained feature extraction model is shown. Figure 6 This diagram illustrates an optional image data processing flow in an image retrieval application scenario. The overall implementation flow of this embodiment can be divided into two stages: model training and model application. The model training stage can be executed by the training server 10, and the model deployment and application stage is executed by the retrieval server 20. The following is a detailed explanation... Figures 3 to 6 The solutions provided in the embodiments of this application are described below.

[0198] Phase 1: Model Training Phase

[0199] This stage can be executed by the training server 10. The training server 10 can iteratively train the neural network model to be trained based on the samples in the training set until the preset conditions are met, thus obtaining a trained feature extraction model. In image retrieval application scenarios, the training set is a set of sample images, which includes a large number of sample images with labels. Each sample image has at least one label, and at least some sample images in the sample image set have at least two labels. That is to say, the sample images can be multi-labeled images.

[0200] The overall training process of the model can mainly include: 1) Data preparation: sample image set, similar sample pairs for metric learning; 2) Preparation of multi-label Fasttext vectors (obtaining the initial semantic vector of each candidate category label); 3) Relevance matrix between candidate category labels; 4) Model forward propagation and model learning.

[0201] The following is combined Figure 6Steps S11 to S15 shown illustrate the implementation process of the model training method in this application scenario. The execution order of steps S12 to S14 can be changed or they can be executed simultaneously.

[0202] Step S11: Obtain the sample image set.

[0203] This application does not limit the method of obtaining sample images. For example, a large number of images can be collected and labeled manually or using a pre-trained image multi-label model to obtain a large number of labeled sample images. For example, a multi-label model can be pre-trained based on an open-source image dataset and used as the labeling model. This model can predict the probability that an image belongs to each category label, and the category labels with probabilities greater than a threshold (the threshold can be adjusted according to the actual situation, such as being set to 0.5) can be used as the image labels. Of course, other existing, directly obtainable multi-label models can also be used as a way to obtain the labels of sample images. For example, a quick way is to use a multi-label model from a business application.

[0204] After obtaining the sample image set, when actually training the model, since the amount of data in the training set is usually quite large, it is generally not possible to process all the sample images at once. The sample image set can be divided into multiple batches, and the images in one batch are used for each training. Assume that there are a total of N sample images in the sample image set, and each batch includes bs sample images.

[0205] Step S12: Based on the labeled images of the samples, determine the similar and dissimilar sample pairs in each batch.

[0206] Similar and dissimilar sample pairs are used to calculate the model's similarity loss (i.e., the first training loss), enabling metric learning of the model. For each batch of sample images, it is necessary to obtain the similar sample pairs within that batch, that is, the similar and dissimilar sample pairs formed by each of the bs sample images and the remaining bs-1 images. Each batch generates a total of bs*(bs-1) / 2 sample pairs. Optionally, sample pairs can be generated in the following way:

[0207] First, the multiple labels of each sample image can be recorded using a one-hot encoding method. For example, suppose there are four candidate category labels and four sample images. The first sample image has two labels, which are the first and second labels of the four candidate categories. The second sample image has the second and third labels, the third sample image has the fourth label, and the fourth sample image has the first, second, and third labels. The labels for these four sample images can then be recorded as follows:

[1100]

[0110] ,

[0001] ,

[1110] Then, the label similarity matrix sim-matrix = Labels * Labels.T is calculated, where Labels represents the label matrix, which includes the label records of each sample image, Labels.T represents the transpose of the matrix, and * represents matrix multiplication. After calculating the sim-matrix, the label similarity (first similarity) between each sample and itself in the sim-matrix is ​​set to 0 to obtain sim-matrix2 (the similarity matrix with self-related components removed). sim-matrix2 records the label similarity between each sample and all other samples outside of that sample, which is the first similarity of each sample pair containing that sample. Taking the above four sample images as an example, the Labels in this example can be represented as follows: Labels = [

[1100] ,

[0110] ,

[0001] ,

[1110] ] Labels.T is as follows: Labels.T = [

[1001] ,

[1101] ,

[0101] ,

[0010] ] Multiplying Labels and Labels.T gives Sim-matrix = [

[2102] ,

[1202] ,

[0010] ,

[2203] ] The first row

[2102] in the Sim-matrix represents the label similarity between the first sample image and the 1st to 4th sample images, which are 2, 1, 0, and 2 respectively. After removing the similarity between a sample and itself, we get

[0102] . That is, the similarity matrix between sample images, Sim-matrix2 = [

[0102] ,

[1002] ,

[0000] ,

[2200] ] From the above sim-matrix2, similar sample pairs can be obtained: such as based on

[0102] of the first sample image, similar sample pairs can be obtained: (sample image 1, sample image 2, similarity weight 1), (sample image 1, sample image 4, similarity weight 2), which can be recorded as (i,j,W) pair [i,j]), where i and j represent two sample images, W pair [i,j] represents the label similarity between the two images, W pair [i,j] = 0 indicates that the two sample images are dissimilar (i.e., the second sample pair) or do not have the same label, W pair [i,j]>0 indicates that the two sample images are similar (i.e., the first sample pair), have the same label, and W pair The larger the value of [i,j], the greater the number of people with the same label. Using the above method, we can obtain similar sample pairs, dissimilar sample pairs, and label similarity between each sample pair based on the labeled images of each sample in each batch. Step S13: Determine the correlation matrix of the labels based on the labeled labels of all sample images in the training set. This step is used to calculate the relevance of each candidate category label to each of the multiple candidate category labels. The relevance matrix is ​​a matrix that records this relevance, also known as the relevance attention value (adj) matrix. Assuming the total number of candidate category labels is Nc = 100, the adj matrix is ​​a 100×100 matrix. Optionally, this matrix can be obtained in the following way: a) For each of the 100 labels, count the number of times the label co-occurs with each of the 100 labels in the training set, and obtain a 100x100 counting matrix adj_count, where the 100 elements in the i-th row of the matrix represent the number of times the i-th label co-occurs with each of the 100 labels. b) For each label, calculate the total number of samples in the training set, label_count. label_count can be recorded as a 100x1 vector, where the i-th value in the vector is the number of times the i-th label appears in all sample images. c) Divide the i-th row of the adj_count matrix by the value of the i-th element of label_count to obtain the normalized co-occurrence matrix adj_norm (the value of the matrix is ​​between 0 and 1), which can also be called the co-occurrence probability of the label. d) Set the values ​​in adj_norm that are less than a set value (such as 0.1) to 0 (that is, if the co-occurrence probability is less than a certain value, the co-occurrence can be considered invalid) to obtain the final co-occurrence adj matrix, which is called the label correlation matrix. Step S14: Obtain the initial semantic vector for each candidate category label. This step generates the lexical embedding (vector representation of words) for each candidate category label, i.e., the initial semantic vector. This can be achieved by extracting feature vectors from the label name of each candidate category label using a text feature model. Alternatively, the Fasttext model can be used to obtain the lexical embedding for each label. Specifically, the Chinese name of each candidate category label can be input into Fasttext, which converts the input label name into a feature vector of a set dimension, such as a 1×300 vector, meaning the initial semantic vector contains 300 feature values. The scheme provided in this application represents each label as a 1×300 embedding, and this embedding contains the semantic information of the label. Compared with representing the label as a 1xNc one-hot vector (Nc represents the total number of labels, and one-hot type labels represent only 0 or 1 values, where 1 indicates that the image contains a certain label and 0 indicates that it does not), the feature representation obtained by the above-mentioned embedding method provided in this application is denser, and it can compress the similarity between labels, which is beneficial to the relevance learning in the subsequent training process. In addition, the representation dimension of the labels obtained by this method does not increase with the number of labels, avoiding the problem that the one-hot method may have too large Nc, resulting in an excessively long one-hot vector that consumes a lot of training resources and causes poor training speed. In this application scenario, assuming there are 100 category labels to be learned, the 100 labels are input into fasttext in sequence, and finally a 100x300 label vocabulary embedding vector is generated. In order to distinguish it from the label embedding (label feature vector) that needs to be learned by the second feature extraction network later, the output of fasttext is called vocabulary embedding, i.e., initial semantic vector. Step S15: Based on the sample image set, the above correlation matrix, and the word embeddings of each candidate category label, iteratively train the neural network model to obtain the trained feature extraction model. Figure 5 The diagram shows a schematic representation of a neural network model provided in an embodiment of this application. Figure 5 As shown, the first feature extraction network in this model includes Figure 5 The convolutional network, fully connected layer Fc1, and hash layer shown, and the second feature extraction network include Figure 5 The relevant learning modules and fully connected layer Fc2 are shown in the diagram. The final feature extraction model applied is the pre-trained first feature extraction network. The model learning process of this neural network model mainly includes: one branch of the model extracts the image embedding through the CNN module, and the other branch obtains the word embeddings of each candidate category label and the correlation matrix adj between each label after FastText embedding based on all labels. Then, the relevant learning module composed of relevant self-attention layers extracts the label embedding for this task. The image embedding and label embedding are mapped to the same feature dimension after passing through mapping layers (Fc1, Fc2). The feature vector obtained by the image embedding after passing through Fc1 is the initial feature vector of the sample image, and the feature vector obtained by the label embedding of the candidate category label after passing through Fc2 is the label feature vector of the label. Then, by passing the initial feature vector of the sample image and the label feature vector of the labeled label of the sample image through the fusion module, the image embedding feature under the control of the label related weight is obtained. Then, the floating-point hash feature (the target feature vector of the sample image in this scenario) is obtained through the hash layer. During training, image label similarity loss (first training loss) and quantization loss (second training loss) can be calculated based on the hash features of sample images. When processing the image to be processed based on the trained feature extraction model, the image embedding of the image to be processed can be obtained through the CNN module mentioned above. The image embedding is then passed through Fc1 and the hash layer to obtain the hash features of the image to be processed. Furthermore, the floating-point hash features can be mapped into a binary hash, i.e., a quantized feature vector, through the sign function, and retrieval can be performed based on this vector. In this application scenario: Assume the vector dimensions of both Fc1 and Fc2 outputs are 256, and there are a total of 100 candidate category labels. During training, Fc2 will produce a 100×256 vector, representing the label feature vectors of the 100 labels. Then, based on the multiple labels of the input sample image, select the labels from all 100 labels present in the image. Extract the label feature vectors of each label from the 100×256 vector and calculate their average. For example, if the sample image has 5 labels, then find the average from the 100×256 vector. The label feature vectors of these 5 labels are 5×256 vectors. An average of these 5 256-dimensional vectors is calculated to obtain a 1×256 vector. This 1×256 vector is then multiplied by a weighted 1×256 vector from the image output by Fc1 (i.e., fusing the initial feature vector of the image with the label feature vectors of the image's annotations) to obtain a 1×256 fused vector. This fused vector is input into a hash layer to obtain a hash representation, i.e., the target feature vector. Further, the hash representation can be quantized to obtain the quantized feature vector corresponding to the sample image. During training, similarity loss and quantization loss are calculated based on the hash features of similar and dissimilar sample pairs. The following is a combination of... Figure 5 The various parts of the neural network model are explained, such as... Figure 5 As shown, the model structure can be divided into 8 parts: a basic image feature CNN module, an image embedding mapping module (fully connected layer Fc1), a label word embedding correlation matrix adj (i.e., the label correlation matrix), and a CNN module. Figure 5 The module includes the relevant value (adj), the relevant learning module, the label embedding mapping module (fully connected layer Fc2), the fusion of two embeddings, the hash layer, and the loss calculation module. It is understandable that... Figure 5 The CNN module, Fc1, feature fusion, and hashing layer shown in the diagram, which extract features from images 1 and 2, actually only have one of the branches in the model. Figure 5 The diagram only illustrates the two branches to help understand the model training process. In the diagram, multi-label 1 represents the labels of image 1, and multi-label 2 represents the labels of image 2. The specific network architecture of the CNN module, Fc1, Fc2, and hash layer in the model is not limited in this embodiment; theoretically, any network architecture that can achieve the corresponding functions is acceptable. The following provides an optional structure for each part: 1) Optionally, the base image feature model, i.e., the CNN, can use the base feature extraction layer of ResNet101 (a residual network), which may include multiple convolutional layers and a pooling layer. The CNN module can also use other structures, such as ResNet18. When an image is input into this base image feature model, its depth features can be obtained. 2) The deep features extracted from the basic image feature model are fed into the image embedding mapping module to generate image embedding mapping. Here, the image embedding mapping module is a single fully connected layer Fc structure. Optionally, this mapping module can also use a network structure of one or more Fc+ReLU activation structures stacked together. In this scenario embodiment, the embedding mapping module takes Fc1 as an example. The output of this module is a 1×256 feature vector, which can be regarded as the initial feature vector of the image. 3) The relevant matrix adj of the tag word embedding. This parameter represents the relevant information between each candidate category tag. The matrix adj does not need to be learned and can be calculated using the method provided above. In this scenario embodiment, the matrix is ​​a 100*100 matrix. 4) Related learning modules ( Figure 5 (Related learning shown in the diagram): Optionally, this module may include one or more cascaded self-attention modules. The following example uses a module with two self-attention modules, i.e., two self-attention layers, each with different input and output dimensions. Optionally, the relevant parameters of the two self-attention layers can be as shown in the table below. The output size of the first layer is Nc×1024, meaning each of the Nc labels has a feature dimension of 1024. The output size of the second layer is Nc×256, allowing the acquisition of a 256-dimensional feature vector for each label. Layer name Output size Layer structure Self-attention 1 Nc×1024 Attention layer Self-attention 2 Nc×256 Attention layer 5) Label embedding mapping module: This module generates label embedding maps. Optionally, the label embedding mapping module can be a single-layer Fc structure, or alternatively, a stack of multiple Fc+ReLU activation structures. This module performs feature mapping on the label feature vectors output by the related learning modules to obtain label feature vectors for each label with the same dimension as the output dimension of the image embedding mapping module. 6) Embedding Fusion Operation: The fusion operation merges the 1×256 image vector output by the image embedding mapping module and the 1×256 label feature vector output by the label embedding mapping module. Optionally, a weighted multiplication fusion method can be used. For a sample image, the 1×256 label feature vector of the image's label output by the label embedding mapping module (if there are multiple labels, the average of the label feature vectors of multiple labels) is used as the weight to weight the 1×256 image vector of the image (the feature values ​​at the same position of the two vectors are multiplied) to generate a new embedding for the image, called joint embedding. 7) Hash layer, optional. This layer can adopt one or more fully connected layers (Fc structure). After performing a cross-learning on the fused joint embedding, it outputs floating-point hash features (target feature vector). The hash features are then processed by the sign function (quantization processing) to generate binary hashes. During training, the training loss is directly calculated using the hash features (since the model directly outputs floating-point hash features, each value of the floating-point output needs to be binarized by the sign function to obtain the vector, which is the final binary hash in practical applications). The following is combined Figure 5 The structure of the neural network model shown below provides a detailed explanation of the training process of this model. 1) Parameter initialization: The correlation matrix adj can be calculated by counting the number of times each candidate category label appears in the sample image set and the number of times the label co-occurs in the sample image set, based on the labeled labels of each sample image in the sample image set. For the basic image feature model, the model parameters can be pre-trained and used as the initialization parameters for this part. For other layers, such as the mapping layer and the related learning layer mentioned above, the model parameters can be initialized using a Gaussian distribution with a variance of 0.01 and a mean of 0. 2) Set the parameters to be learned: parameters of the basic image feature model, parameters of the image embedding mapping module, parameters of the related learning module, parameters of the label embedding mapping module, and parameters of the hash layer. 3) Learning rate: Optional. A learning rate of lr = 0.0005 can be used. After every 10 iterations, lr can be reduced to 0.1 times the original value. 4) Learning process: Perform epoch iterations on the full dataset (all sample images in the sample image set); process the full dataset once per iteration (one batch of data is used in each training process, and all batches are trained once to complete one iteration) until the average loss no longer decreases under a certain epoch (the model's loss function converges) or a certain set number of training epochs are reached. 5) The specific operations in each iteration of each epoch are as follows: Divide the total N samples into Nb = N / bs batches, with each batch consisting of bs samples. For each batch, determine the similar and dissimilar sample pairs in the batch using the data preparation method described above, and then perform the following operations: ① Model forward pass: Set all parameters of the model to the state to be learned. During training, the neural network model performs forward calculation on an input sample image to obtain the hash result, denoted by Q. Q is a 1x256 vector representing the hash feature of the sample image. ② Calculation of total training loss: For each sample pair, calculate the similarity loss between the hash features of the two sample images in the pair; for each sample image, calculate the coding loss (or quantization loss). The weighted sum of the average of these two losses is the total loss. The specific calculation methods for these two losses will be introduced later. ③ Model parameter update: Stochastic gradient descent (SGD) is used to calculate the gradient backwards of the loss in step ② to obtain the updated values ​​of all model parameters, and then the network is updated to adjust the parameters that the model needs to learn. For each training process, the specific processing flow of the neural network model for the sample images is as follows: Each sample image (e.g.) Figure 5Images 1 and 2 are output to the CNN to obtain the image's depth features. These features are mapped by Fc1 to obtain the image's initial feature vector. The 100 class labels (all labels) are processed by FastText to obtain 100 label feature dimensions of 300 word embeddings, which are the initial semantic vectors. The 100×300 word embeddings and the correlation matrix of the labels are input into the relevance learning module. The 100 labels are processed by this module to obtain the feature vectors (the second feature vector obtained from the last feature update operation mentioned above). These 100 processed feature vectors are mapped by Fc2 to obtain 100 label embeddings. For each sample image, the initial feature vector of the image obtained by Fc1 mapping is fused with the label embedding of the labeled label of the image in the 100 labels by the fusion module. The fused features are then input into the hash layer to obtain the hash features of the image. The hash features can be quantized using the sign function to obtain the quantized feature vector. Figure 7 and Figure 8 This diagram illustrates the model structure of the relevant learning module in this embodiment. Figure 7 This is the structure of the first self-attention layer. Figure 8 This is the structure of the second attention layer, with two attention layers stacked together. Figure 7 and Figure 8 In the model parameters w and b, each attention layer has a weight matrix w (weight) and a bias b (bias). The first attention layer has a weight matrix of size input-dim × h-dim and a bias b of size 1 × h-dim. Here, input-dim is the dimension of the input features, which is the dimension of the word embeddings of 100 candidate category labels, i.e., 300. h-dim is the mapping dimension, which is the dimension of the hidden features of each label output by this attention layer. The second attention layer has a weight matrix of size h-dim × out-dim and a bias b of size 1 × h-dim. The first attention layer takes as input the 100×300 word embeddings and the 100×100 relevance matrix adj from Fasttext. Its output size is 100×h_dim, representing the hidden feature vectors (second feature vectors) of 100 labels with dimension h_dim. The second attention layer takes as input the 100×h_dim output from the first attention layer and the relevance matrix adj. Its output size is 100×out_dim, representing the feature vectors of the 100 labels input to Fc2. In this embodiment, out_dim is 256. The operation of each self-attention layer is as follows: like Figure 7 and Figure 8 As shown, input 1 of each attention layer is abbreviated as input, and input 2 is abbreviated as adj. First, input is multiplied by the weight matrix w, thus mapping the 100 embedding dimensions of input to 100 hidden features of h_dim (hidden dimension). In the matrix multiplication, each value in the hidden dimension is obtained by weighted fusion of all 300 dimensions of the input embedding. Therefore, M1 is the hidden feature result obtained from the first feature cross. Then, input 2, i.e., the correlation matrix adj, is used to multiply the hidden features of M1 after autocorrelation removal (1-adj) to obtain M2. Finally, the bias is added to obtain the output result M3 of the attention layer. The calculation principle can be expressed as follows: M1 = input × w M2=(1-adj)×M1 M3 = M2 × b The difference between the inputs of the first and second attention layers lies in input 1. The first attention layer's input 1 is the output vector of the Fasttext corresponding to 100 labels, while the second attention layer's input 1 is the 100×h_dim hidden feature vector output from the first attention layer. Through the above operations, the 100×out_dim output of the second attention layer is obtained. Figure 5 After the Labelembedding shown, Fc2 can produce a 100×256 output, which is the label feature vector of 100 labels. After obtaining the 256-dimensional label feature vectors for 100 labels, the outputs of Fc1 for each sample image can be fused as described above, and then mapped through a hash layer to obtain the hash features of the sample images. After obtaining the hash features of each sample image, the total training loss of the model can be calculated using the following expression: L. total : Where w1 and w2 represent the weights of the first training loss (similarity loss) and the second training loss (quantization loss), Ns1 represents the number of similar sample pairs S1 in a batch, Ns2 represents the number of dissimilar sample pairs S2 in a batch, and L sim [x] represents the similarity (1-L) of the hash features of two samples in sample pair x among Ns1 similar sample pairs. sim [x] represents the degree of difference), W pair[i, j] represents the label similarity weight, with higher weights for greater similarity. This represents the label similarity between two samples i and j in sample pair x (i.e., the first similarity, which is the number of identical labels the two samples share). L sim [y] represents the similarity of the hash features of two samples in sample pair y among Ns2 dissimilar sample pairs, L coding [p] represents the quantization loss of the sample image p. Optionally, the above similarity L sim [x]、L sim [y] can be obtained by calculating the vector inner product of the hash features of the two samples, as shown in the following formula. The similarity value ranges from 0 to 1, H i and H j H represents the hash feature of two sample images, ||H i ||2 represents H i The 2-norm. In this embodiment of the application, for any similar sample pair, W pair [i, j] > 0, hoping that Hi and Hj have the same distribution, that is, the similarity between hash features is close to 1. In loss calculation (Ltotal formula), we hope that 1-L sim The closer to 0 or the smaller the better; multiply by W in Ltotal. pair [i, j] is taken into account that the more times i and j have the same number of labels, the better W becomes. pair The larger [i, j] is, the more it needs to be compared to W in order to satisfy the learning strategy of Ltotal approaching 0 or being as small as possible. pair Smaller 1-L when [i, j] = 1 sim The effect of [i, j] is that it is better than W. pair When [i, j] = 1, the larger L is... sim [i, j], thus achieving the effect of adjusting the loss based on the label differences (number of identical labels, number of different labels) between two samples, so that the hash representation has a higher similarity when there are more co-occurring labels. For dissimilar samples, W pair [i, j] = 0, hoping L sim The closer to 0 or the smaller the better, so Lsim can be directly used as the loss for dissimilar sample pairs in Ltotal. For each sample image, the quantization loss L coding[p] Optionally, in this scenario embodiment, the sign function can be used to quantize the feature values ​​in the hash features of each sample image into feature values ​​of 1 or -1. The quantization loss is used to evaluate whether the quantization result of the sample image is close enough to 1 or -1. Since the final application after quantization needs to be mapped to the two values ​​of -1 or 1, in application scenarios that require the use of quantized feature vectors, if each bit of the hash feature Q of the sample image is close enough to 1 or -1, then the above triplet metric loss (similarity loss) is sufficient to describe the application of quantized features. Otherwise, even if the triplet metric effect is good, it does not mean that the quantized representation in the application is good enough (this is because in the actual application scenario of using quantized feature vectors, the hash layer output of the image to be processed will be quantized once by sign before application. The hash feature output from the model is a floating-point number, while the application is a quantized binary feature, which will have quantization error. For this scenario, if quantization loss is not added during training, if the quantization error is very large, it will affect the representation effect and lead to unsatisfactory image retrieval effect). Therefore, it is desirable that each bit of Q is close enough to 1 or -1. Optionally, the quantization loss for each sample image can be calculated using the following expression: Among them, Q i Let B be the feature value of the i-th bit in the hash feature Q of the sample image. In this scenario embodiment, the output dimension of the hash layer is 256. i For Q i The quantized feature values ​​are the quantization target, B i By Q i The target code for the quantization learning task is generated through the sign function (in practical applications, other quantization functions can also be selected according to application requirements), that is, by using the sign function to generate the target code for the quantization feature Q. i The target code B is calculated using the sign function. i Finally, the target encoding of Q is B. Then, the regression loss (the formula for calculating the quantization loss Lcoding mentioned above) can be used to calculate the L2 distance between the hash feature Q and the target encoding B, thus obtaining the quantization loss of the sample image. Through continuous training, the hash feature Q of the sample image can become increasingly closer to the target encoding B, i.e., the L2 distance becomes smaller and smaller. For each training session, after calculating the total training loss L... total Then, based on the total loss, the gradient descent algorithm can be used to update the parameters that need to be learned in the model, and the model with updated parameters can continue to be trained until a model that meets the conditions is obtained. The first feature extraction network in the model that meets the conditions is used as the trained feature extraction model, that is... Figure 5 The CNN module, Fc1 layer, and hash layer shown can be deployed to the retrieval server 20, which can then use the model to achieve accurate image retrieval. Phase Two: Model Application Phase like Figure 6 As shown, the model application stage can include steps S21 to S23, namely, receiving the retrieval request sent by the terminal, extracting features from the retrieval image in the retrieval request using a feature extraction model to obtain the feature vector of the retrieval image, searching for matching images in the image database based on this feature vector to obtain the retrieval result, and returning the retrieval result to the user terminal. Specifically, after obtaining the feature vector (hash feature) of the retrieval image through the trained feature extraction model, the hash feature is quantized, and image retrieval is performed based on the quantized feature vector to improve retrieval efficiency. Optionally, a pre-trained feature extraction model can be used to extract features from all stock images (all retrieved images in the image database) to obtain a hash feature Q for each retrieved image. Q is then transformed using the sign function to obtain a binary vector for each retrieved image and stored in the database. For the query image (i.e., the retrieved image), the feature extraction model extracts the Q of the image and binarizes it to obtain a binary vector. The binary vector of the query image is compared one by one with the binary vectors of each stored stock image. The similarity between the query image and each stock image can be obtained by calculating the Hamming distance between the binary vectors of the query image and each stock image. Calculating the Hamming distance for binary vectors can accelerate computation. After calculating the Hamming distance for each stock image, the top k (i.e., topk) stock images with distances from the top k (from smallest to largest) can be selected as matching images (i.e., search results) based on the distances in ascending order, or stock images with distances less than a set distance can be returned as matching images. This allows for the retrieval of stock images that are semantically similar to the query image. Figure 4 The diagram illustrates another image retrieval method applicable to embodiments of this application. To improve retrieval efficiency, the retrieval server 20 can first use a trained feature extraction model to construct a retrieval index (index system construction) for the images in the image database. Then, the quantized feature vector of the retrieved image and the constructed index system are used to perform the retrieval. The steps for image retrieval based on the hash features output by the model can be as follows: 1) Obtain the hash features of the stock images through the model. This step is used to extract the hash features of each retrieved image in the image database through the model. 2) Obtain the quantization center of the hash feature. After obtaining the hash features of each retrieved image in the image database, the quantization features of each retrieved image can be obtained through a quantization function. By clustering the quantization features of all retrieved images in the image database, several classes (i.e., clusters) can be obtained. For each class, the cluster center (which represents the center vector of all hash features in a class, and can be regarded as the quantization center) of the class can be calculated based on the quantization features of each retrieved image belonging to that class. For example, the quantization features of all retrieved images can be clustered into Nt cluster centers using k-means clustering (a commonly used clustering method). 3) Use Nt quantization centers as the index for retrieval. Figure 4 The retrieval system 1 shown is used for bucket retrieval, and establishes the association between the index and the image database (quantization center and quantization features of each retrieved image in the image data). 4) During the retrieval process, based on the quantitative characteristics of the query image (i.e., the image to be retrieved in the retrieval request), find the nearest (i.e., the most similar) set number of indices, and use the retrieved images associated with these indices as candidate recall images. Figure 4 The recall output in the process. 5) Based on the quantization features of the candidate recall images and the quantization features of the query images, calculate the Euclidean distance between the query images and each candidate recall image, sort the candidate recall images in ascending order of distance, and take the top K candidate recall images after sorting as the retrieval results, i.e., the final recall results. Understandably, in practical applications, if retrieval efficiency is not a concern, image retrieval can be performed directly based on hash features. For example, clustering can be performed based on hash features to obtain several cluster centers. Each cluster center can be used as an index. The similarity between each retrieved image and each cluster center can be calculated based on the hash features of the retrieved image. One or more cluster centers with high similarity can be found. Then, the retrieval results are obtained by calculating the similarity between the hash features of each retrieved image belonging to these cluster centers and the hash features of the retrieved image. Correspondingly, during training, operations related to quantization loss can be omitted. Alternatively, the feature extraction model can also be without a hash layer. The vector mapped by Fc1 can be used as the feature vector of the image. Correspondingly, during training, the fused features can be directly used as the target feature vector of the sample image. The model training method provided in this application allows the image's hash features to fully learn the representational adjustments in the image embedding caused by label correlation during model training (fusing the feature vectors of the image's labeled labels into the image's hash features). Through the correlation learning model, the model can learn the irrelevant representations of each label in the feature space after removing the correlation between labels. Therefore, through continuous training, redundant information of related labels can be suppressed, and differential information can be expressed, improving the discriminative power of each label's feature vector. This fuses the label feature vectors of each sample image's labeled labels with the image's feature vectors, enabling the model to learn a more discriminative final image feature representation. This improves the performance of the finally trained feature extraction model and better meets application requirements. It will be clear to those skilled in the art that the specific structure of the neural network structure provided and used in the embodiments of this application may include, but is not limited to, the structures provided above. For example, the basic image feature model in the first feature extraction network may use a ResNet network structure or other backbone networks. Furthermore, when using a pre-trained multi-label acquisition model to obtain sample labels, the multi-label acquisition model may use a ResNet101-based model structure, or a model structure such as Swin-Transformer (a transformer model with sliding window operation and hierarchical design). It may be pre-trained using open-source training datasets such as ImageNet or Open Image, or it may use a multi-label model used in business applications. During model training, the initialization method of the network's model parameters can use any existing model parameter initialization method. The preset parameters in the model's loss function and the adjustment method of the model parameters can be adjusted according to actual needs. Taking image processing scenarios as an example, the optional implementation schemes provided in this application can have the following advantages: 1) It can learn feature representations (such as hash features) that differentiate the similarity of multi-label images: By leveraging the rich multi-label semantic correlation, i.e. label similarity, it provides information on the commonalities and differences between images with common labels when representing images, so that the similarity of two images with different labels (both with the same label and different labels) can be distinguished, avoiding the problem of low distinguishability between images with some co-occurring labels. 2) Utilize label relevance information to avoid feature redundancy among co-occurring labels and improve the information expression effect of differential labels: Optionally, the relevance matrix can be used to learn and correct in the relevance learning network to generate the relevance relationship between multiple labels. Based on the relevance relationship, the lexical embedding of the label, i.e. the initial semantic vector, is adjusted and a label embedding output is generated, so that the label relevance is easier to learn. Finally, the label embedding and the image embedding are fused to output a more expressive final image representation. 3) Applying the feature extraction model trained based on the training method provided in the embodiments of this application to the data retrieval scenario can effectively improve the accuracy and recall of retrieval: by using the multi-label semantic space to supplement the hash space of metric learning, the model’s feature expression effect on co-occurring labels and differential labels is effectively improved, thereby achieving more accurate recall. 4) By incorporating the homogeneity and commonality of labels into model learning, it is possible to achieve the effect that the hash features of sample images with more identical labels have greater similarity, thereby achieving the distinguishability of co-occurring label similarity. 5) It can remove feature redundancy caused by multi-label correlation: By leveraging self-attention correlation learning and suppressing excessive repetitive correlation information in the representation, redundant information between two semantically similar labels can be removed, thereby allowing for improved representation of irrelevant information of unrelated labels, and ultimately achieving more effective learning of image representation. Corresponding to the data processing method provided in the embodiments of this application, the embodiments of this application also provide a data processing apparatus, such as... Figure 9 As shown, the data processing device 100 may include a data acquisition module 110 and a data processing module 120. The data acquisition module 110 is used to acquire data to be processed, wherein the data to be processed is data containing information of at least one modality in multimedia information; The data processing module 120 is used to extract feature vectors from the data to be processed through a trained feature extraction model, determine the category of the data to be processed based on the feature vectors, or determine the target data that matches the data to be processed from the dataset. The feature extraction model is obtained by repeatedly training a neural network model on multiple samples with at least one labeled label. The labeled label is the true class label of the sample, and each labeled label is one of multiple candidate class labels. The neural network model includes a first feature extraction network and a second feature extraction network. The feature extraction model is the first feature extraction network that meets the training termination condition. During training, the label feature vector of each candidate category label is obtained through the second feature extraction network, the initial feature vector of each sample is obtained through the first feature extraction network, and the target feature vector of the sample is obtained by fusing the initial feature vector of the sample with the label feature vectors of each labeled label of the sample. The total training loss of the neural network model is determined based on the labeled label and target feature vector of each sample. Optionally, the aforementioned feature extraction model can be obtained by a model training device performing training operations on a neural network model, wherein the model training device can be used for: The training operation is repeatedly performed on the neural network model to be trained based on the training set (i.e., the multiple samples mentioned above) until a preset condition is met. The neural network model includes a first feature extraction network and a second feature extraction network. The first feature extraction network that meets the preset condition is taken as the trained feature extraction model. The training operation includes the following steps: The label feature vector of each candidate category label is obtained through the second feature extraction network; For each sample, the initial feature vector of the sample is obtained through the first feature extraction network, and the target feature vector of the sample is obtained by fusing the initial feature vector with the label feature vectors of each labeled label of the sample. Based on the labeled tags and target feature vectors of each sample, the total training loss of the neural network model is determined. If the preset conditions are not met, the model parameters of the neural network model will be adjusted based on the total training loss. Optionally, the above training process also includes: obtaining the initial semantic vector of each candidate category label; determining the number of co-occurrences between each pair of labels in the multiple candidate category labels based on the labeled labels of multiple samples; for each candidate category label, determining the correlation between the label and each label in the multiple candidate category labels based on the number of co-occurrences between the label and each label in the multiple candidate category labels, wherein the correlation between two labels characterizes the co-occurrence probability between the two labels; The label feature vector for each candidate category label is obtained in the following way: Based on the initial semantic vectors of each candidate category label and the correlation between each candidate category label and each label among multiple candidate category labels, the label feature vector of each candidate category label is extracted through the second feature extraction network. Optionally, the label feature vector for each candidate category label is obtained in the following way: For each candidate category label, determine the non-relevance between the label and each of the multiple candidate category labels based on the relevance between the label and each of the multiple candidate category labels; The second feature extraction network performs at least one feature update operation to obtain a label feature vector for each candidate category label based on the second feature vector of each candidate category label obtained from the last feature update operation; wherein the feature update operation includes the following steps: Based on the semantic correlation between the initial semantic vectors of each candidate category label, the first feature vector corresponding to each candidate category label is obtained by feature fusion of the initial semantic vectors of each candidate category label. For each candidate category label, the irrelevance between the label and each of the multiple candidate category labels is used as the weight. The first feature vectors of each of the multiple candidate category labels are weighted and fused. The second feature vector of the label is obtained based on the weighted and fused feature vector. The second feature vector is used as the initial semantic vector of the label in the next feature update operation. Optionally, the relevance between each candidate category label and each of the multiple candidate category labels is obtained in the following way: Based on the labeled labels of multiple samples, determine the number of times each candidate category label appears in multiple samples; For each candidate category label, the ratio of the number of times the label co-occurs with each of the multiple candidate category labels to the number of times the label appears is used as the co-occurrence probability between the label and each of the multiple candidate category labels. The correlation between the label and each of the multiple candidate category labels is obtained based on the co-occurrence probability. Optionally, for each sample, the target feature vector of that sample is obtained in the following way: If the sample has only one label, then the label feature vector of that label is used as the weight vector of the initial feature vector of the sample. The initial feature vector of the sample is weighted using this weight vector. Based on the weighted feature vector, the target feature vector of the sample is obtained. If the sample has multiple labels, the label feature vectors of the multiple labels of the sample are fused together, and the fused label feature vector is used as the weight vector of the initial feature vector of the sample. The initial feature vector of the sample is weighted using the weight vector, and the target feature vector of the sample is obtained based on the weighted feature vector. Optionally, the target feature vector for each sample is obtained in the following way: For each sample, the first feature extraction network performs the following operations to obtain the target feature vector of the sample: Extract the initial feature vector of the sample; The initial feature vector of the sample and the label feature vectors of each labeled label of the sample are fused together, and feature extraction is performed on the fused feature vector to obtain the target feature vector of the sample. Optionally, the above training operation further includes: quantizing the target feature vector of the sample to obtain the quantized feature vector corresponding to the sample; The total training loss of the neural network model is obtained in the following way: Based on the labeled tags and target feature vectors of each sample, the first training loss is determined; based on the difference between the target feature vector and the quantized feature vector of each sample, the second training loss is determined; based on the first training loss and the second training loss, the total training loss of the neural network model is determined. Optionally, the total training loss of the neural network model is determined in the following way: For each sample pair in a plurality of samples, the first similarity of the sample pair is determined based on the number of identical labeled tags that the two samples in the sample pair have. Based on the first similarity of each sample pair, determine the first sample pair and the second sample pair in each sample pair, wherein the first similarity of the first sample pair is greater than the first similarity of the second sample pair; For each first sample pair, the training loss corresponding to the first sample pair is obtained based on the difference between the target feature vectors of the two samples in the first sample pair; For each second sample pair, the training loss corresponding to the second sample pair is determined based on the second similarity between the target feature vectors of the two samples in the second sample pair. The total training loss of the neural network model is obtained based on the training loss corresponding to each first sample pair and the training loss corresponding to each second sample pair. Optionally, for each first sample pair, the training loss corresponding to that first sample pair is obtained in the following way: Determine the degree of difference between the target feature vectors of the two samples in the first sample pair; Based on the first similarity of the first sample pair, the loss weight of the first sample pair is determined, wherein the loss weight is not less than 1, and the loss weight of each first sample pair is positively correlated with the first similarity of the first sample pair. The first similarity of the first sample pair is used as the weight, and the difference of the first sample pair is weighted to obtain the training loss of the sample pair. It is understood that the apparatus of this application embodiment can execute the method provided in this application embodiment, and the implementation principle is similar. The actions performed by each module in the apparatus of each embodiment of this application correspond to the steps in the method of each embodiment of this application. For detailed functional descriptions of each module of the apparatus, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here. This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program stored in the memory, it can implement the method in any optional embodiment of this application. Figure 10 A schematic diagram of the structure of an electronic device to which an embodiment of the present invention applies is shown, such as... Figure 10 As shown, the electronic device can be a server or a user terminal, and it can be used to implement the methods provided in any embodiment of the present invention. like Figure 10 As shown, the electronic device 2000 may primarily include at least one processor 2001. Figure 10 The diagram shows components such as a memory 2002, a communication module 2003, and an input / output interface 2004. Optionally, these components can be connected and communicate with each other via a bus 2005. It should be noted that... Figure 10 The structure of the electronic device 2000 shown is merely illustrative and does not constitute a limitation on the electronic devices to which the methods provided in the embodiments of this application are applicable. The memory 2002 can be used to store operating systems and applications, etc. The applications can include computer programs that implement the methods shown in the embodiments of the present invention when invoked by the processor 2001, and can also include programs for implementing other functions or services. The memory 2002 can be ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices that can store information and computer programs, or it can be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disk storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto. Processor 2001 is connected to memory 2002 via bus 2005, and implements corresponding functions by calling application programs stored in memory 2002. Processor 2001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 2001 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. Electronic device 2000 can connect to a network via communication module 2003 (which may include, but is not limited to, components such as a network interface) to communicate with other devices (such as user terminals or servers) through the network and achieve data interaction, such as sending data to or receiving data from other devices. Communication module 2003 may include wired network interfaces and / or wireless network interfaces, meaning the communication module may include at least one of wired or wireless communication modules. Electronic device 2000 can connect to required input / output devices, such as keyboards and display devices, via input / output interface 2004. Electronic device 2000 itself may have a display device, and other display devices can also be connected externally via interface 2004. Optionally, storage devices, such as hard drives, can also be connected via interface 2004 to store data from electronic device 2000, retrieve data from storage devices, or store data from storage devices into memory 2002. It is understood that input / output interface 2004 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to input / output interface 2004 can be a component of electronic device 2000 or an external device connected to electronic device 2000 when needed. The bus 2005 used to connect the various components may include a pathway for transmitting information between these components. The bus 2005 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Depending on its function, the bus 2005 can be divided into address bus, data bus, control bus, etc. Optionally, for the solution provided in the embodiments of the present invention, the memory 2002 can be used to store a computer program that executes the solution of the present invention, and the processor 2001 runs the computer program. When the processor 2001 runs the computer program, it implements the operation of the method or apparatus provided in the embodiments of the present invention. Based on the same principle as the method provided in the embodiments of this application, the embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the corresponding content of the aforementioned method embodiments. This application also provides a computer program product, which includes a computer program that, when executed by a processor, can implement the corresponding content of the aforementioned method embodiments. It should be noted that the terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text. It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this. The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Acquire data to be processed, wherein the data to be processed is data containing information of at least one modality in multimedia information; The feature vector of the data to be processed is extracted using a trained feature extraction model; Based on the feature vector of the data to be processed, determine the category of the data to be processed, or identify target data that matches the data to be processed from the dataset; The feature extraction model is obtained by repeatedly training a neural network model based on multiple samples. Each of the multiple samples is a sample with at least one labeled label. The labeled label is the true category label of the sample. Each labeled label is one of multiple candidate category labels. The neural network model includes a first feature extraction network and a second feature extraction network. The feature extraction model is a first feature extraction network that meets the training termination condition. During training, the label feature vector of each candidate category label is obtained through the second feature extraction network, the initial feature vector of each sample is obtained through the first feature extraction network, and the target feature vector of the sample is obtained by fusing the initial feature vector of the sample with the label feature vectors of each labeled label of the sample. The total training loss of the neural network model is determined based on the labeled labels and target feature vectors of each sample. The label feature vector of each candidate category label is obtained in the following way: Obtain the initial semantic vector of each candidate category label; For each candidate category label, the correlation and non-correlation between the label and each of the plurality of candidate category labels are determined based on the number of times the label co-occurs with each of the plurality of candidate category labels. The correlation between two labels characterizes the co-occurrence probability between the two labels. At least one feature update operation is performed through the second feature extraction network, and a label feature vector for each candidate category label is obtained based on the second feature vector of each candidate category label obtained from the last feature update operation. The feature update operation includes the following steps: Based on the semantic correlation between the initial semantic vectors of each candidate category label, a first feature vector corresponding to each candidate category label is obtained by performing feature fusion on the initial semantic vectors of each candidate category label. For each candidate category label, the irrelevance between the label and each of the multiple candidate category labels is used as a weight to perform a weighted fusion of the first feature vectors of each of the multiple candidate category labels. The second feature vector of the label is obtained based on the weighted fused feature vector, and the second feature vector is used as the initial semantic vector of the label in the next feature update operation.

2. The method according to claim 1, characterized in that, The semantic correlation between the initial semantic vectors of each candidate category label is used to obtain a first feature vector corresponding to each candidate category label by performing feature fusion on the initial semantic vectors of each candidate category label, including: For each candidate category label, an attention mechanism is used to determine the weight of each candidate category label corresponding to the candidate category label based on the semantic relevance between the initial semantic vector of the candidate category label and the initial semantic vector of each candidate category label in the multiple candidate category labels. Based on the weight of each candidate category label corresponding to the candidate category label, the initial semantic vector of each candidate category label in the multiple candidate category labels is weighted and fused to obtain the first feature vector of the candidate category label.

3. The method according to claim 2, characterized in that, For each candidate category label, determining the relevance and non-relevance between the label and each of the plurality of candidate category labels based on the number of times the label co-occurs with each of the plurality of candidate category labels includes: Based on the labeled tags of the multiple samples, determine the number of times each candidate category tag appears in the multiple samples; For each candidate category label, the ratio of the number of times the label co-occurs with each of the multiple candidate category labels to the number of times the label appears is used as the co-occurrence probability between the label and each of the multiple candidate category labels. The correlation between the label and each of the multiple candidate category labels is obtained based on the co-occurrence probability. For each candidate category label, the non-correlation between the label and each of the plurality of candidate category labels is determined based on the correlation between the label and each of the plurality of candidate category labels.

4. The method according to claim 1, characterized in that, For each sample, the target feature vector is obtained in the following way: If the sample has only one label, then the label feature vector of that label is used as the weight vector of the initial feature vector of the sample. The initial feature vector of the sample is weighted using this weight vector. Based on the weighted feature vector, the target feature vector of the sample is obtained. If the sample has multiple labels, the label feature vectors of the multiple labels of the sample are fused together, and the fused label feature vector is used as the weight vector of the initial feature vector of the sample. The initial feature vector of the sample is weighted using the weight vector, and the target feature vector of the sample is obtained based on the weighted feature vector.

5. The method according to claim 1, characterized in that, For each sample, the target feature vector is obtained in the following way: For each sample, the first feature extraction network performs the following operations to obtain the target feature vector of the sample: Extract the initial feature vector of the sample; The initial feature vector of the sample and the label feature vectors of each labeled label of the sample are fused together, and feature extraction is performed on the fused feature vector to obtain the target feature vector of the sample.

6. The method according to claim 1, characterized in that, For each of the samples, the training operation further includes: The target feature vector of the sample is quantized to obtain the quantized feature vector corresponding to the sample. The total training loss of the neural network model is determined in the following way: The first training loss is determined based on the labeled tags and target feature vectors of each sample. The second training loss is determined based on the difference between the target feature vector and the quantized feature vector of each sample. The total training loss of the neural network model is determined based on the first training loss and the second training loss.

7. The method according to any one of claims 1 to 6, characterized in that, The total training loss of the neural network model is determined in the following way: For each sample pair among the plurality of samples, a first similarity is determined based on the number of identical labels shared by the two samples in the sample pair, wherein each sample pair consists of two samples among the plurality of samples; Based on the first similarity of each sample pair, a first sample pair and a second sample pair are determined in each sample pair, wherein the first similarity of the first sample pair is greater than the first similarity of the second sample pair; For each of the first sample pairs, the training loss corresponding to the first sample pair is obtained based on the difference between the target feature vectors of the two samples in the first sample pair; For each second sample pair, the training loss corresponding to the second sample pair is determined based on the second similarity between the target feature vectors of the two samples in the second sample pair; The total training loss of the neural network model is obtained based on the training loss corresponding to each of the first sample pairs and the training loss corresponding to each of the second sample pairs.

8. The method according to claim 7, characterized in that, For each first sample pair, the step of obtaining the training loss corresponding to the first sample pair based on the difference between the target feature vectors of the two samples in the first sample pair includes: Determine the degree of difference between the target feature vectors of the two samples in the first sample pair; Based on the first similarity corresponding to the first sample pair, the loss weight corresponding to the first sample pair is determined, wherein the loss weight is not less than 1, and the loss weight corresponding to each first sample pair is positively correlated with the first similarity corresponding to the first sample pair. The first similarity of the first sample pair is used as the weight, and the difference of the first sample pair is weighted to obtain the training loss of the sample pair.

9. A data processing apparatus, characterized in that, include: A data acquisition module is used to acquire data to be processed, wherein the data to be processed is data containing information of at least one modality in multimedia information; The data processing module is used to extract feature vectors from the data to be processed using a trained feature extraction model, and to determine the category of the data to be processed based on the feature vectors, or to determine target data that matches the data to be processed from the dataset. The feature extraction model is obtained by repeatedly training a neural network model based on multiple samples. Each of the multiple samples is a sample with at least one labeled label. The labeled label is the true category label of the sample. Each labeled label is one of multiple candidate category labels. The neural network model includes a first feature extraction network and a second feature extraction network. The feature extraction model is a first feature extraction network that meets the training termination condition. During training, the label feature vector of each candidate category label is obtained through the second feature extraction network, the initial feature vector of each sample is obtained through the first feature extraction network, and the target feature vector of the sample is obtained by fusing the initial feature vector of the sample with the label feature vectors of each labeled label of the sample. The total training loss of the neural network model is determined based on the labeled labels and target feature vectors of each sample. The label feature vector of each candidate category label is obtained in the following way: Obtain the initial semantic vector of each candidate category label; For each candidate category label, the correlation and non-correlation between the label and each of the plurality of candidate category labels are determined based on the number of times the label co-occurs with each of the plurality of candidate category labels. The correlation between two labels characterizes the co-occurrence probability between the two labels. At least one feature update operation is performed through the second feature extraction network, and a label feature vector for each candidate category label is obtained based on the second feature vector of each candidate category label obtained from the last feature update operation. The feature update operation includes the following steps: Based on the semantic correlation between the initial semantic vectors of each candidate category label, a first feature vector corresponding to each candidate category label is obtained by performing feature fusion on the initial semantic vectors of each candidate category label. For each candidate category label, the irrelevance between the label and each of the multiple candidate category labels is used as a weight to perform a weighted fusion of the first feature vectors of each of the multiple candidate category labels. The second feature vector of the label is obtained based on the weighted fused feature vector, and the second feature vector is used as the initial semantic vector of the label in the next feature update operation.

10. The apparatus according to claim 9, characterized in that, The first feature vector corresponding to each candidate category label is obtained in the following way: For each candidate category label, an attention mechanism is used to determine the weight of each candidate category label corresponding to the candidate category label based on the semantic relevance between the initial semantic vector of the candidate category label and the initial semantic vector of each candidate category label in the multiple candidate category labels. Based on the weight of each candidate category label corresponding to the candidate category label, the initial semantic vector of each candidate category label in the multiple candidate category labels is weighted and fused to obtain the first feature vector of the candidate category label.

11. The apparatus according to claim 10, characterized in that, The correlation and non-correlation between each candidate category label and each of the plurality of candidate category labels are obtained in the following way: Based on the labeled tags of the multiple samples, determine the number of times each candidate category tag appears in the multiple samples; For each candidate category label, the ratio of the number of times the label co-occurs with each of the multiple candidate category labels to the number of times the label appears is used as the co-occurrence probability between the label and each of the multiple candidate category labels. The correlation between the label and each of the multiple candidate category labels is obtained based on the co-occurrence probability. For each candidate category label, the non-correlation between the label and each of the plurality of candidate category labels is determined based on the correlation between the label and each of the plurality of candidate category labels.

12. The apparatus according to claim 9, characterized in that, For each sample, the target feature vector is obtained in the following way: If the sample has only one label, then the label feature vector of that label is used as the weight vector of the initial feature vector of the sample. The initial feature vector of the sample is weighted using this weight vector. Based on the weighted feature vector, the target feature vector of the sample is obtained. If the sample has multiple labels, the label feature vectors of the multiple labels of the sample are fused together, and the fused label feature vector is used as the weight vector of the initial feature vector of the sample. The initial feature vector of the sample is weighted using the weight vector, and the target feature vector of the sample is obtained based on the weighted feature vector.

13. The apparatus according to claim 9, characterized in that, For each sample, the target feature vector is obtained in the following way: For each sample, the first feature extraction network performs the following operations to obtain the target feature vector of the sample: Extract the initial feature vector of the sample; The initial feature vector of the sample and the label feature vectors of each labeled label of the sample are fused together, and feature extraction is performed on the fused feature vector to obtain the target feature vector of the sample.

14. The apparatus according to claim 9, characterized in that, For each of the samples, the training operation further includes: The target feature vector of the sample is quantized to obtain the quantized feature vector corresponding to the sample. The total training loss of the neural network model is determined in the following way: The first training loss is determined based on the labeled tags and target feature vectors of each sample. The second training loss is determined based on the difference between the target feature vector and the quantized feature vector of each sample. The total training loss of the neural network model is determined based on the first training loss and the second training loss.

15. The apparatus according to any one of claims 9-14, characterized in that, The total training loss of the neural network model is determined in the following way: For each sample pair among the plurality of samples, a first similarity is determined based on the number of identical labels shared by the two samples in the sample pair, wherein each sample pair consists of two samples among the plurality of samples; Based on the first similarity of each sample pair, a first sample pair and a second sample pair are determined in each sample pair, wherein the first similarity of the first sample pair is greater than the first similarity of the second sample pair; For each of the first sample pairs, the training loss corresponding to the first sample pair is obtained based on the difference between the target feature vectors of the two samples in the first sample pair; For each second sample pair, the training loss corresponding to the second sample pair is determined based on the second similarity between the target feature vectors of the two samples in the second sample pair; The total training loss of the neural network model is obtained based on the training loss corresponding to each of the first sample pairs and the training loss corresponding to each of the second sample pairs.

16. The apparatus according to claim 15, characterized in that, For each first sample pair, the step of obtaining the training loss corresponding to the first sample pair based on the difference between the target feature vectors of the two samples in the first sample pair includes: Determine the degree of difference between the target feature vectors of the two samples in the first sample pair; Based on the first similarity corresponding to the first sample pair, the loss weight corresponding to the first sample pair is determined, wherein the loss weight is not less than 1, and the loss weight corresponding to each first sample pair is positively correlated with the first similarity corresponding to the first sample pair. The first similarity of the first sample pair is used as the weight, and the difference of the first sample pair is weighted to obtain the training loss of the sample pair.

17. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the method of any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

19. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images