Searching method, device and equipment and computer readable storage medium

Abstract

The application provides a retrieval method, device and equipment and a computer readable storage medium. The method comprises the following steps: in response to a received retrieval request, obtaining retrieval reference information and a trained feature extraction model, wherein a loss function in a training process of the trained feature extraction model at least comprises a first loss function component capable of measuring distance repulsion of multi-label irrelevant sample information; performing feature extraction on the retrieval reference information by using the trained feature extraction model to obtain a retrieval reference vector; determining at least one target object from a retrieval database based on the retrieval reference vector; and outputting the at least one target object. According to the application, the retrieval accuracy can be improved.

Description

Technical Field

[0001] This application relates to Internet technology, and more particularly to a retrieval method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the widespread adoption of internet technology and terminal devices, massive amounts of semantically rich multimedia data have emerged in various internet application scenarios. How to effectively retrieve this vast amount of multimedia data has become a hot research topic in the field of multimedia processing. In retrieval technology, the ability to efficiently and accurately extract feature vectors of retrieval reference information has become a crucial factor affecting retrieval accuracy. Summary of the Invention

[0003] This application provides a retrieval method, apparatus, and computer-readable storage medium that can improve retrieval accuracy.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a retrieval method, including:

[0006] In response to a received retrieval request, retrieval reference information and a trained feature extraction model are obtained. The loss function of the trained feature extraction model during the training process includes at least a first loss function component that can measure distance exclusion of multi-label irrelevant sample information.

[0007] The trained feature extraction model is used to extract features from the retrieval reference information to obtain a retrieval reference vector.

[0008] At least one target object is determined from the retrieval database based on the retrieval reference vector;

[0009] Output the at least one target object.

[0010] This application provides a retrieval device, including:

[0011] The first acquisition module is used to respond to the received retrieval request and acquire retrieval reference information and a trained feature extraction model. The loss function of the trained feature extraction model during the training process includes at least a first loss function component that can measure distance exclusion of multi-label irrelevant sample information.

[0012] The first processing module is used to extract features from the retrieval reference information using the trained feature extraction model to obtain a retrieval reference vector.

[0013] The first determining module is used to determine at least one target object from the retrieval database based on the retrieval reference vector;

[0014] The first output module is used to output the at least one target object.

[0015] This application provides an apparatus, including:

[0016] Memory, used to store executable instructions;

[0017] A processor, when executing executable instructions stored in the memory, implements the method provided in the embodiments of this application.

[0018] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the method provided in this application.

[0019] This application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method provided in this application.

[0020] The embodiments of this application have the following beneficial effects:

[0021] Upon receiving a retrieval request, in response to the request, the system first acquires retrieval reference information and a trained feature extraction model. Then, it uses the trained feature extraction model to extract features from the retrieval reference information to obtain a retrieval reference vector. Based on the retrieval reference vector, it determines at least one target object from the retrieval database and finally outputs the at least one target object. In this embodiment, the loss function of the trained feature extraction model during training includes at least a first loss function component that can exclude multi-label irrelevant sample information by measuring distance. Therefore, it can ensure that the feature vectors between irrelevant objects extracted by the trained feature extraction model are far apart in the metric space, thereby improving the retrieval accuracy when using the feature vector for retrieval. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the network architecture of the retrieval system provided in the embodiments of this application;

[0023] Figure 2 This is a schematic diagram of the structure of the server 400 provided in the embodiment of this application;

[0024] Figure 3 This is a schematic diagram illustrating an implementation flow of the retrieval method provided in an embodiment of this application;

[0025] Figure 4 A schematic diagram illustrating the training process of the feature extraction model provided in this application embodiment;

[0026] Figure 5A schematic diagram illustrating another implementation of the retrieval method provided in this application embodiment;

[0027] Figure 6 This is a schematic diagram illustrating another implementation flow of the retrieval method provided in the embodiments of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0030] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0032] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0033] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0034] 1) A metric space is a set containing all samples, and the distance between any elements in the set is definable; it is also called a distance space.

[0035] 2) Metric learning is a method for learning a measure of the distance distribution between data objects. Its goal is to achieve clustering by minimizing the distance between similar objects and distinguishing dissimilar objects by maximizing the distance between them.

[0036] 3) Content-based image retrieval refers to image retrieval technology that uses the semantic content of images, such as color, texture, and layout features, for analysis and searching.

[0037] 4) Pairwise loss refers to a loss function calculated based on the distance between samples.

[0038] 5) Class proxies refer to a set of class centers randomly assigned before training. During training, samples belonging to a certain class will gradually converge to the vicinity of the class proxies of that class to achieve the effect of clustering.

[0039] 6) Proxy loss refers to the loss function calculated based on the distance between the sample and the class proxy.

[0040] To better understand the retrieval method provided in the embodiments of this application, the retrieval methods in related technologies, as well as the loss functions and shortcomings of the feature extraction models involved in the retrieval methods in related technologies, are explained.

[0041] I. Paired loss based methods.

[0042] By calculating the distance similarity between the feature vectors of two or more samples at each step, the gradient descent direction is determined. This reduces the distance between similar samples to achieve clustering, and increases the distance between dissimilar samples to achieve differentiation. In challenging multi-label retrieval tasks, pairwise loss-based methods can fully explore the deep attribute correlations between samples and estimate semantic distance based on the ranked labels.

[0043] II. Methods based on agency loss.

[0044] In implementation, a set of class proxy vectors is pre-defined as cluster centers for each class, and a corresponding class proxy is randomly assigned to each class before training. During training, the gradient descent direction is determined by calculating the distance similarity between the feature vector of a sample and all class proxy vectors. Samples belonging to a certain class will converge to the space near the corresponding class proxy to achieve the clustering effect, while increasing the distance between samples of this class and other class proxies to achieve the differentiation effect.

[0045] The following explains the disadvantages of the two methods mentioned above in the relevant technologies.

[0046] Pairwise loss-based methods effectively mine the correlation between data and labels, demonstrating satisfactory performance. However, their training complexity typically reaches the square or cube of the number of image data points, resulting in significant time overhead on large-scale image datasets. Furthermore, pairwise loss only optimizes the metric distance for samples within the same batch entering the neural network training, achieving metric optimization for that batch. However, local metric optimization often leads to conflicts in a global sense; that is, satisfying the optimality of samples within a batch may cause conflicts with samples outside the batch, making convergence difficult during training, resulting in unstable clustering effects, and further increasing training time overhead.

[0047] Methods based on proxy loss lack information about the degree of label correlation between samples, and the constructed metric space is difficult to express the complex mapping relationship between feature vectors and label sets in multi-label tasks.

[0048] Methods based on proxy loss consider the distance metric between samples and class proxies, resulting in faster training and convergence speeds and better clustering performance. However, in multi-label scenarios, they lack label information indicating different levels of similarity between samples, making it impossible to reasonably map and rank multi-label samples, thus affecting retrieval performance. Specifically, this manifests as insufficient robustness in the quantity and quality of segmented regions in the metric space.

[0049] Proxy loss distributes similar samples in a space near their corresponding proxy class, forming an approximate hypersphere. However, the generalization ability of a hypersphere in space is weak, and the number of segmented spatial regions is limited, failing to satisfy the mapping relationships of complex label sets in multi-label scenarios. Furthermore, proxy loss simply maps samples with multiple labels to the middle position of their corresponding proxy class. Due to the lack of relevant information between samples, proxy loss cannot exclude dissimilar samples with multiple labels that are close in distance, leading to confusion of irrelevant samples and thus degrading performance. The reasons for these shortcomings of proxy loss are explained below.

[0050] Under agent loss supervision, similar samples will form an approximate hypersphere distribution around the corresponding agent class, and the upper limit of the number of regions Ω(K,C) that can be divided by C hyperspheres in K-dimensional space can be obtained by formula (1-1):

[0051] (1-1);

[0052] Since a dataset with C categories can theoretically have a maximum of 2 C There are different labels, and when C>K+1, there is formula (1-2):

[0053] (1-2);

[0054] Therefore, the hypersphere metric space cannot be partitioned into a sufficient number of spaces to satisfy the mapping relationship between the label set and the metric space. As a result, samples from multiple label sets will be mapped to the same region in the space, causing confusion and leading to performance degradation.

[0055] This application provides a retrieval method, apparatus, device, and computer-readable storage medium that can improve retrieval accuracy. The following describes exemplary applications of the electronic devices provided in this application. These electronic devices can be implemented as various types of user terminals such as laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or as servers. The following describes exemplary applications when the device is implemented as a server.

[0056] See Figure 1 , Figure 1 This is a schematic diagram of the network architecture of the retrieval system 100 provided in the embodiments of this application, as shown below. Figure 1 As shown, the system includes: terminal 200, network 300, server 400 and retrieval database 500. Terminal 200 is connected to server 400 through network 300. Network 300 can be a wide area network or a local area network, or a combination of both.

[0057] The retrieval method described in this application can be applied to various scenarios, such as browser image retrieval, video retrieval, music retrieval, etc. The terminal 200 can have various application clients installed, such as game applications, shopping applications, video viewing applications, music playback applications, etc. These applications can be applications that require downloading and installation, or they can be instant-use applications (applets); this application embodiment does not limit this.

[0058] When a terminal receives a search instruction, it sends a search request to the server in response to the instruction. The search request carries search reference information, which can be an image, video, or audio. After receiving the search request, the server obtains the search reference information and a trained feature extraction model. Using the trained feature extraction model, the server extracts features from the search reference information to obtain a search reference vector. The server 400 determines the similarity between the search reference vector and the binarized feature vectors of each search object in the search database 500. Based on these similarities, the server identifies at least one target object from the multiple search objects and sends this target object to the terminal 200. The terminal 200 then displays the target objects on its display device.

[0059] In the embodiments of this application, the loss function of the trained feature extraction model during the training process includes at least a first loss function component that can exclude multi-label irrelevant sample information by measuring distance. Therefore, it can be guaranteed that the feature vectors between irrelevant objects extracted by the trained feature extraction model are far apart in the metric space, thereby improving the retrieval accuracy when using the feature vector for retrieval.

[0060] In some embodiments, server 400 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 200 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, in-vehicle smart terminal, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0061] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the server 400 provided in an embodiment of this application. Figure 2 The server 400 shown includes at least one processor 410, at least one network interface 420, a bus system 430, and memory 440. The various components in the server 400 are coupled together via the bus system 430. It is understood that the bus system 430 is used to implement communication between these components. In addition to a data bus, the bus system 430 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 430.

[0062] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0063] The memory 440 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 440 may optionally include one or more storage devices physically located away from the processor 410.

[0064] The memory 440 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 440 described in this application embodiment is intended to include any suitable type of memory.

[0065] In some embodiments, memory 440 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0066] Operating system 441 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0067] The network communication module 442 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0068] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A retrieval device 443 stored in memory 440 is shown. This device can be software in the form of programs or plug-ins, and includes the following software modules: a first acquisition module 4431, a first processing module 4432, a first determination module 4433, and a first output module 4434. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0069] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the update method of the search model provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0070] The retrieval method provided in this application will be described in conjunction with exemplary applications and implementations of the server provided in the embodiments of this application.

[0071] This application provides a retrieval method applied to an electronic device, which can be a terminal or a server. This application uses an electronic device as a server as an example for illustration. Figure 3 This is a schematic diagram of an implementation flow of the retrieval method provided in this application embodiment. The following will be combined with... Figure 3 The retrieval method provided in the embodiments of this application will be described.

[0072] Step S101: In response to the received retrieval request, obtain retrieval reference information and the trained feature extraction model.

[0073] In this embodiment of the application, the retrieval request may be sent by the terminal to the server after receiving the retrieval instruction. The retrieval request may carry retrieval reference information. When this step is implemented, after receiving the retrieval request, the server parses the retrieval request and obtains the retrieval reference information. The retrieval reference information may be an image, a video, or an audio. In this embodiment of the application, the type of retrieval reference information is not limited.

[0074] The feature extraction model is a neural network model, which in practical applications can be either a deep learning neural network model or a convolutional neural network model. For image or video retrieval, convolutional neural networks are highly adaptable to images and can significantly reduce the number of model parameters through weight sharing. Therefore, convolutional neural network models are widely used in the field of computer vision. Furthermore, training a convolutional neural network model can learn deep features with better feature representation capabilities.

[0075] The loss function of the trained feature extraction model during the training process includes at least a first loss function component that can exclude multi-label irrelevant sample information by metric distance. In this embodiment, the first loss function component is the cosine similarity loss of multi-label irrelevant samples. Since the first loss function component can rank irrelevant samples by metric distance, the trained feature extraction model can learn the ability to exclude feature vectors between irrelevant objects in the metric space.

[0076] Step S102: Use the trained feature extraction model to extract features from the retrieval reference information to obtain a retrieval reference vector.

[0077] A trained feature extraction model can include an input layer, convolutional layers, pooling layers, fully connected layers, and an output layer. In implementation, this step can begin by preprocessing the retrieval reference information using the input layer of the trained model. This ensures the feature extraction model has standardized input data and that prominent features appear at the center of the convolutional kernels. Then, convolutional layers perform convolution operations on the input data to enhance features and reduce noise. Next, pooling layers pool the convolution results to retain useful feature information and prevent overfitting. Finally, fully connected layers integrate the obtained local features into a global feature, resulting in the retrieval reference vector to be output. The output layer then outputs this detection reference vector.

[0078] Step S103: Determine at least one target object from the retrieval database based on the retrieval reference vector.

[0079] Because the retrieval reference vectors extracted using the trained feature extraction model have high dimensions, such as 128 or 256 dimensions, and the feature values ​​of the feature vectors are usually floating-point numbers, the processing complexity is too high when storing the retrieval reference vectors or using them to determine target objects from the database. Therefore, in this embodiment, the retrieval database stores the binarized feature vectors of the retrieval objects. This step involves first binarizing the retrieval reference vectors to obtain binarized reference vectors, then calculating the distances between the binarized reference vectors and each binarized feature vector, and finally determining at least one target object based on these distances.

[0080] Step S104: Output the at least one target object.

[0081] When this step is implemented by the server, outputting at least one target object means sending at least one target object to the terminal corresponding to the retrieval request. In some embodiments, after receiving at least one target object, the terminal displays the at least one target object on its own display device.

[0082] In some embodiments, when the step is performed by a terminal, outputting at least one target object means presenting the at least one target object on the terminal's own display device.

[0083] In the retrieval method provided in this application embodiment, after receiving a retrieval request, in response to the retrieval request, retrieval reference information and a trained feature extraction model are first obtained. Then, the trained feature extraction model is used to extract features from the retrieval reference information to obtain a retrieval reference vector. Based on the retrieval reference vector, at least one target object is determined from the retrieval database. Finally, the at least one target object is output. In this application embodiment, the loss function of the trained feature extraction model during the training process includes at least a first loss function component that can exclude multi-label irrelevant sample information by metric distance. Therefore, it can be ensured that the feature vectors between irrelevant objects extracted by the trained feature extraction model are far apart in the metric space, thereby improving the retrieval accuracy when using the feature vector for retrieval.

[0084] In some embodiments, prior to step S101, the following methods may also be used: Figure 4 The steps shown are used to train the feature extraction model to obtain a trained feature extraction model. The following is a combination of... Figure 4 The training process of the feature extraction model is explained.

[0085] Step S001: Obtain the preset feature extraction model and training dataset.

[0086] The preset feature extraction model uses default parameters. The training dataset includes multiple training samples and their label vectors. Training samples can be training images, videos, or audio. The label vector of a training sample can have a dimension of 1*P, where P is the total number of types. Assuming P is 4, for an image, the labels can be plant, animal, sky, and snow, and the label vector will have a dimension of 1*4. Assuming the four label values ​​correspond to plant, animal, sky, and snow respectively, when an image's labels include plant, animal, and sky, its label vector is [1,1,1,0]. When an image's labels include plant and sky, its label vector is [1,0,1,0].

[0087] Step S002: Obtain the proxy vectors for each class and the number of samples N required for each training of the model.

[0088] In this step, the values ​​of each element in the obtained class proxy vector are set to default values. During the training of the feature extraction model, in addition to adjusting the model parameters, the values ​​of each element in the class proxy vector are also adjusted. The number of samples N required for each model training session is an integer greater than 1. For example, N can be 100, meaning that 100 training samples will be obtained from the training dataset each time the model is trained as the target training samples for this training session.

[0089] Step S003: Determine whether the training termination conditions have been met.

[0090] In implementation, the training termination condition can be determined by the number of iterations or by the loss value, which will be explained below.

[0091] 1. Determine by the number of iterations.

[0092] First, the number of training iterations is obtained. Then, it is determined whether the number of training iterations has reached a preset threshold. When the number of training iterations reaches the preset threshold, it is determined that the training has ended. When the number of training iterations has not reached the preset threshold, it is determined that the training has not ended.

[0093] Second, judge by the loss value.

[0094] First, obtain the total loss value of the loss function after the last training. Then, determine whether the total loss value is less than a preset loss threshold. If the total loss value is less than the preset loss threshold, it is determined that the training termination condition has been met. If the total loss value is greater than or equal to the loss threshold, it is determined that the training termination condition has been met.

[0095] In this embodiment of the application, when it is determined that the training end condition has not been met, the process proceeds to step S004; when it is determined that the training end condition has been met, the process proceeds to step S007.

[0096] Step S004: Use the feature extraction model to obtain N target training samples from the training dataset, and use the feature extraction model to extract features from each target training sample to obtain the predicted feature vector of each target training sample.

[0097] In this step, N target training samples are randomly selected from the training dataset. Then, each target training sample is preprocessed through the input layer of the feature extraction model. Next, the convolutional layer performs convolution operations on the preprocessed target training samples to obtain the convolution result. Then, the pooling layer and fully connected layer perform pooling and fully connected processing on the convolution result to finally obtain the predicted feature vector of each target training sample.

[0098] Step S005: Obtain the loss function of the feature extraction model.

[0099] In this embodiment, the loss function includes a first loss function component capable of metrically rejecting multi-label irrelevant sample information and a second loss function component for clustering training samples. In practical implementation, the loss function of the feature extraction model can be expressed as: ,in, For loss function, For the first loss function component, For the second loss function component, This is a hyperparameter.

[0100] Step S006: The feature extraction model is trained using the loss function, the predicted feature vectors of each target training sample, the label vectors of each target training sample, and the class proxy vectors.

[0101] In this step, firstly, based on the label vectors of each target training sample, at least one multi-label irrelevant sample pair is determined. Then, a first loss value is determined based on the first loss function component of the loss function and the predicted feature vectors of the target training samples in each multi-label irrelevant sample pair. A second loss value is then determined based on the second loss function component of the loss function, the predicted feature vectors of each target training sample, the label vectors of each target training sample, and each class proxy vector. Finally, a total loss value is determined based on the first and second loss values. Afterward, the feature extraction model is trained by backpropagation based on the loss values ​​to adjust the parameters of the feature extraction model.

[0102] After step S006, proceed to step S003 again, and repeat steps S003 to S006.

[0103] Step S007: Determine the obtained trained feature extraction model.

[0104] Through steps S001 to S007, the training process of the feature extraction model is completed. The training process of the feature extraction model includes a first loss function component that can measure distance rejection of multi-label irrelevant sample information and a second loss function component that is used to cluster training samples. By introducing a small number of irrelevant samples, the proxy loss is alleviated. In multi-label scenarios, it is impossible to reject dissimilar samples with multiple labels that are close in distance, which would lead to the confusion of irrelevant samples. Thus, the accuracy of the extracted feature vector can be improved without affecting the overall training speed and convergence.

[0105] In some embodiments, step S006 above, "training the feature extraction model using the loss function, the predicted feature vectors of each target training sample, the label vectors of each target training sample, and each class proxy vector," can be achieved through the following steps:

[0106] Step S0061: Based on the label vectors of each target training sample, determine at least one multi-label irrelevant sample pair.

[0107] In this step, firstly, based on the label vectors of each target training sample, it is determined whether the i-th target training sample and the j-th target training sample have any common label. When it is determined that the i-th target training sample and the j-th target training sample do not have any common label, the i-th target training sample and the j-th target training sample are determined as a multi-label irrelevant sample pair, where i=1,2,…N, j=1,2,…N, and i is not equal to j.

[0108] To determine whether the i-th and j-th target training samples have any identical labels, we can first determine the inner product of the label vectors of the i-th and j-th target training samples, and then check if the inner product is 0. If the inner product is 0, we determine that the i-th and j-th target training samples do not have any identical labels; if the inner product is not 0, we determine that the i-th and j-th target training samples have at least one identical label.

[0109] Step S0062: Determine the first loss value based on the first loss function component of the loss function and the predicted feature vector of the target training sample in each multi-label irrelevant sample pair.

[0110] In implementing this step, the total number of multi-label irrelevant sample pairs is first determined. Then, based on the predicted feature vectors of the two target training samples in each multi-label irrelevant sample pair, the first negative sample loss value of each multi-label irrelevant sample pair is determined, and the total first negative sample loss value is determined. Then, the total first negative sample loss value is divided by the total number of pairs to obtain the first average negative sample loss value. In this embodiment, the first average negative sample loss value is determined as the first loss value.

[0111] Step S0063: Determine the second loss value based on the second loss function component of the loss function, the predicted feature vector of each target training sample, the label vector of each target training sample, and each class proxy vector.

[0112] In this step, firstly, based on the label vectors of each target training sample and the class proxy vectors, the target positive training sample and target negative training sample corresponding to each class proxy vector are determined; then, based on the class proxy vectors and the predicted feature vectors of the target positive training samples corresponding to each class proxy vector, a first positive sample loss value is determined, and the first positive sample loss value is divided by the total number of positive samples to obtain a first average positive sample loss value; then, based on the class proxy vectors and the predicted feature vectors of the target negative training samples corresponding to each class proxy vector, a second negative sample loss value is determined, and then the second negative sample loss value is divided by the total number of negative samples to obtain a second average negative sample loss value; finally, the sum of the first average positive sample loss value and the second average negative sample loss value is determined as the second loss value.

[0113] Step S0064: Determine the total loss value based on the first loss value and the second loss value.

[0114] In implementation, the first loss value and hyperparameters will be used. The product of the first and second loss values, plus the second loss value, gives the total loss value.

[0115] Step S0065: Perform backpropagation training on the feature extraction model based on the total loss value to adjust the parameters of the feature extraction model.

[0116] In practical applications, the gradient descent method can be used to backpropagate and train the feature extraction model based on the loss value in order to adjust the parameters of the feature extraction model.

[0117] In steps S0061 to S0065 above, firstly, multi-label irrelevant sample pairs are identified, and then the first loss value and the second loss value are determined based on the first loss function component and the second loss function component, respectively. Then, the total loss value is determined. Since the first loss function component can constrain the mutual exclusion between irrelevant samples, when the feature extraction model is backpropagated and trained based on the total loss value to adjust the parameters of the feature extraction model, the feature extraction model can learn the ability to exclude irrelevant samples from each other in the metric space.

[0118] In some embodiments, step S0062 above, "determining the first loss value based on the first loss function component of the loss function and the predicted feature vector of the target training sample in each multi-label irrelevant sample pair," can be implemented through steps 621 to 624 below. Each step is described below. These steps include:

[0119] Step 621: Obtain the total number of multi-label unrelated sample pairs.

[0120] Step 622: Based on the predicted feature vectors of the two target training samples in each multi-label irrelevant sample pair, determine the first negative sample loss value of each multi-label irrelevant sample pair.

[0121] The first negative sample loss value for each multi-label irrelevant sample pair can be the cosine similarity loss value. In implementation, the first negative sample loss value for each multi-label irrelevant sample pair can be determined using formula (1-3):

[0122] (1-3);

[0123] in, and Let be the predicted feature vector of two target training samples in a multi-label unrelated sample pair. The preset interval factor is determined based on the number of categories and the number of hash bits, and is a real number between -1 and 1.

[0124] Step 623: Determine the first average negative sample loss value based on the first negative sample loss value of each multi-label irrelevant sample pair and the total number of logs.

[0125] Here, the first negative sample loss values ​​of each multi-label irrelevant sample pair are summed to obtain the first negative sample total loss value. Then, the first negative sample total loss value is divided by the total logarithm to obtain the first average negative sample loss value.

[0126] Step 624: Determine the first average negative sample loss value as the first loss value.

[0127] In some embodiments, the above step S0063, "determining the second loss value based on the second loss function component of the loss function, the predicted feature vector of each target training sample, the label vector of each target training sample, and each class proxy vector", can be implemented by the following steps 631 to 636, and each step is described below.

[0128] Step 631: Based on the label vectors of each target training sample and the class proxy vectors, determine the target positive training sample and the target negative training sample corresponding to each class proxy vector.

[0129] In this embodiment of the application, each label category corresponds to a class proxy vector. When implementing this step, if the value of the kth label is determined to be 1 based on the label vector of the target training sample, it indicates that the target training sample is a positive sample of the class proxy vector corresponding to the kth label category; if the value of the kth label is determined to be 0, it indicates that the target training sample is a negative sample of the class proxy vector corresponding to the kth label category.

[0130] Suppose that the label vector of a certain target training sample is [1,1,0,0], and the first to fourth labels correspond to plants, animals, sky, and snow respectively. Then the target training sample is the target positive training sample of the class proxy vector corresponding to plants, the target positive training sample of the class proxy vector corresponding to animals, the target negative training sample of the class proxy vector corresponding to sky, and the target negative training sample of the class proxy vector corresponding to snow.

[0131] Step 632: Determine the total number of positive samples of the target positive training samples corresponding to each class of proxy vector and the total number of negative samples of the target negative training samples corresponding to each class of proxy vector.

[0132] Step 633: Determine the first positive sample loss value based on the predicted feature vectors of each class of proxy vectors and the target positive training samples corresponding to each class of proxy vectors.

[0133] Class proxy vector The target positive training samples corresponding to this type of proxy vector The loss value between the predicted feature vectors can be determined using formula (2-2) in the following embodiment:

[0134] (2-2);

[0135] In this embodiment of the application, the loss values ​​between each class of proxy vector and the predicted feature vector of the target positive training sample corresponding to each class of proxy vector are summed to obtain the first positive sample loss value.

[0136] Step 634: Determine the second negative sample loss value based on the predicted feature vectors of the target negative training samples corresponding to each class of proxy vectors.

[0137] Here, class proxy vector The target negative training samples corresponding to this type of proxy vector The loss value between the predicted feature vectors can be determined using formula (2-3) in the following embodiment:

[0138] (2-3);

[0139] In this embodiment, the loss values ​​between each class of proxy vector and the predicted feature vector of the target negative training sample corresponding to each class of proxy vector are summed to obtain the second negative sample loss value.

[0140] Step 635: Determine a first average positive sample loss value based on the first positive sample loss value and the total number of positive samples, and determine a second average negative sample loss value based on the second negative sample loss value and the total number of negative samples.

[0141] Dividing the first positive sample loss value by the total number of positive samples yields the first average positive sample loss value, and dividing the first negative sample loss value by the total number of negative samples yields the second average negative sample loss value.

[0142] Step 636: The sum of the first average positive sample loss value and the second average negative sample loss value is determined as the second loss value.

[0143] In some embodiments, step S007 may further evaluate the performance metrics of the trained feature extraction model through the following steps:

[0144] Step S008: Obtain test data.

[0145] The test data includes multiple test samples and label vectors for each test sample.

[0146] Step S009: Use the trained feature extraction model to extract features from each test sample to obtain the predicted feature vector of each test sample.

[0147] Step S010: Determine the prediction information of each test sample based on the prediction feature vector of each test sample, and determine the evaluation index value of the trained feature extraction model based on the prediction information of each test sample and the corresponding label vector.

[0148] In this step, the predicted feature vectors of each test sample are first binarized to obtain the binarized prediction vectors of the test samples. Then, using the binarized prediction vectors and category proxy vectors of the test samples, the prediction information of each test sample is determined. Finally, based on the prediction information of each test sample and the corresponding label vector, the evaluation metric value of the trained feature extraction model is determined. This evaluation metric value can be precision, recall, etc.

[0149] Step S011: Determine that the evaluation index value is less than the preset index threshold, and then obtain multiple target training samples again.

[0150] If the evaluation index value is less than the preset index threshold, it means that the performance index of the trained feature extraction model has not met the standard and training needs to continue. At this time, multiple target training samples are obtained again.

[0151] In some embodiments, when the evaluation index value is greater than or equal to the index threshold, it indicates that the performance index of the trained feature extraction model has met the standard, and there is no need to continue training the trained feature extraction model.

[0152] Step S012: The trained feature extraction model is further trained using the multiple target training samples until the evaluation index value of the retrained feature extraction model reaches the index threshold.

[0153] Through the above steps S007 to S012, after the initial training of the feature extraction model is obtained, the performance index of the initial training feature extraction model can be evaluated to obtain the evaluation index value. If the evaluation index value is determined to be below the preset index threshold, it means that the performance index of the initial training feature extraction model has not met the standard. At this time, new training data needs to be obtained again and the feature extraction model needs to be trained again until the evaluation index of the trained feature extraction model reaches the index threshold. This can ensure the accuracy of feature extraction when using the trained feature extraction model.

[0154] Based on the above embodiments, this application further provides a retrieval method, applied to... Figure 1 The network architecture shown is Figure 5 This is a schematic diagram illustrating another implementation of the retrieval method provided in the embodiments of this application, which is described below in conjunction with... Figure 5 The retrieval method provided in the embodiments of this application will be described.

[0155] Step S201: The server obtains the trained feature extraction model.

[0156] The loss function used in the training process of the trained feature extraction model includes a first loss function component that measures the distance of multi-label irrelevant sample information to exclude them, and a second loss function component that is used to cluster the training samples.

[0157] Step S202: The server obtains object information of each search object in the search database.

[0158] Retrieve various types of objects stored in the database, such as images, text, videos, and audio.

[0159] In step S203, the server uses the trained feature extraction model to extract features from the information of each object, and obtains the object feature vector corresponding to each retrieved object.

[0160] The implementation process of this step is similar to that of step S102, and the implementation process of step S102 can be referred to when implementing it.

[0161] Step S204: The server performs binarization processing on the object feature vectors corresponding to each search object to obtain the binarized feature vectors of each search object.

[0162] In implementation, this step can be achieved by using a deep hash function to binarize the object feature vector, transforming the object feature vector in the original space into its representation in Hamming space, resulting in a binarized feature vector. The binarized feature vector has a lower dimensionality than the initial object feature vector, and its value is either 0 or 1, requiring less storage space than multi-bit floating-point numbers.

[0163] In some embodiments, the binarization module that performs binarization processing can be used as an intermediate layer of the feature extraction model. After obtaining the feature vector of the retrieval object, the binarization module is used to binarize the feature vector so that the final output of the feature extraction model is the binarized feature vector.

[0164] In step S205, the server stores the binarized feature vectors of each search object into the search database.

[0165] In step S206, the terminal launches the application client and selects an image as the reference information for retrieval.

[0166] In this embodiment, the application client can be a browser client, or a shopping client, video client, etc. An information search entry control is provided on the application client's display page, through which users can input text, voice, images, etc., to search for reference information. In this embodiment, taking an image as the search reference information is used as an example. The image can be stored in the terminal's local space, and can be captured using the terminal's built-in image acquisition device, downloaded from the network, or sent to the terminal by another terminal. This embodiment does not limit the source of the image.

[0167] In step S207, the terminal receives a search instruction and sends a search request to the server in response to the search instruction.

[0168] In this embodiment of the application, the search request carries an image as search reference information.

[0169] In step S208, the server responds to the received search request and obtains search reference information.

[0170] During implementation, the server parses the retrieval request to obtain retrieval reference information.

[0171] In step S209, the server uses the trained feature extraction model to extract features from the retrieval reference information to obtain the retrieval reference vector.

[0172] The implementation process of this step is similar to that of step S102, and the implementation process of step S102 can be referred to when implementing it.

[0173] In step S210, the server performs binarization processing on the retrieval reference vector to obtain a binarized reference vector of the retrieval reference information.

[0174] The implementation process of this step is similar to that of step S204. In implementation, a deep hash function can be used to binarize the retrieval reference vector, transforming the retrieval reference information in the original space into an expression in Hamming space, thus obtaining a binarized reference vector.

[0175] Step S211: The server determines the distances between the binarized reference vector and the binarized feature vectors of each of the retrieved objects.

[0176] This step, when implemented, determines the Hamming distances between the binarized reference vector and the binarized feature vectors of each retrieved object. A larger distance between the binarized reference vector and the binarized feature vector of the retrieved object indicates a lower similarity between the retrieved reference information and the retrieved object.

[0177] In step S212, the server identifies the search objects whose distances are less than the distance threshold as the target objects.

[0178] When the distance between the binary feature vector of a search object and the binary reference vector of the search reference information is less than a preset distance threshold, it means that the similarity between the search reference information and the search object meets the matching condition, and the search object is then identified as the target object.

[0179] In step S213, the server sends at least one target object to the terminal.

[0180] In implementation, the server can include at least one target object in the retrieval response and send the response to the terminal. Before sending, the server will sort the at least one target object in ascending order of its distance from the retrieval reference information and then send the sorted at least one target object to the terminal.

[0181] Step S214: The terminal displays at least one target object on its own display device.

[0182] When the terminal displays at least one target object, it also displays them in ascending order of distance from the search reference information. That is, the target object with higher similarity to the search reference information is displayed earlier.

[0183] In the retrieval method provided in this application embodiment, the server first uses a trained feature extraction model to extract features from the retrieval objects in the retrieval database, and performs binarization processing on the extracted object feature vectors to obtain binarized feature vectors. The server then stores the binarized feature vectors of each retrieval object in the database. This not only reduces the storage space required for storing the feature vectors of the retrieval objects, but also, after receiving the retrieval instruction, the terminal sends a retrieval request carrying retrieval reference information to the server. The server first uses the trained feature extraction model to extract features from the retrieval reference information to obtain retrieval reference vectors. Similarly, it also performs binarization processing on the detection reference vectors to obtain binarized reference vectors. Therefore, when determining the similarity between the retrieval reference information and each retrieval object, the distance between the binarized reference vector and the binarized feature vector of each retrieval object is determined, thereby reducing computational complexity and improving computational efficiency. Then, based on these distances, the server determines at least one target object from multiple retrieval objects and sends the at least one target object to the terminal for display on the terminal's display device. Because the trained feature extraction model utilizes a loss function that includes constraints on the mutual exclusion of irrelevant samples during training, it ensures that the feature vectors of irrelevant search objects extracted by the trained model are mutually exclusive in the metric space. This ensures that when determining the distance between the feature vectors of the retrieval reference information and each search object, the distance between the retrieval reference information and irrelevant search objects is significantly different, thus avoiding confusion caused by high similarity between the retrieval reference information and irrelevant search objects, and ultimately improving the accuracy of the search results.

[0184] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0185] This application provides a retrieval method. Figure 6 This is a schematic diagram illustrating the implementation process of model training and image retrieval provided in the embodiments of this application. The following is in conjunction with... Figure 6 The model training and image retrieval processes provided in the embodiments of this application are described.

[0186] Step S601: Obtain training data.

[0187] Step S602: Use a feature extraction model to extract features from the training data to obtain a training feature vector.

[0188] Step S603: During the training phase, the loss value is determined using the training feature vector and the mixed loss function.

[0189] Step S604: Backpropagate the loss value to the feature extraction model to update the network parameters.

[0190] Step S605: In the inference phase, the reference feature vector of the retrieved image is extracted using the trained feature extraction model.

[0191] Step S606: Binarize the reference feature vector of the retrieved image to obtain a binarized reference vector.

[0192] Step S607: Determine the search results from the search database based on the binarized reference vector.

[0193] In the training phase of the feature extraction model, to address the poor generalization of surrogate loss, this application proposes a hybrid loss function based on surrogate and pairwise loss. This hybrid loss function is used to train the feature extraction model. The idea is to introduce a small amount of key, multi-labeled, irrelevant sample pair information to solve the surrogate loss problem without affecting the overall training speed and convergence, thereby improving performance. This hybrid loss function includes a surrogate loss function and an irrelevant sample pair loss function, which will be described separately below.

[0194] a) Proxy loss function

[0195] In this embodiment, linear baseline surrogate loss is used to cluster the samples. The surrogate loss is calculated using the following formula:

[0196] (2-1);

[0197] in, D M For the training set, P C For class proxy collection, M The number of samples in the training set. C Let be the number of categories, and when the i-th sample contains the j-th label. The value is 1, therefore, This represents the total number of positive samples. When the i-th sample does not contain the j-th label... The value is 1, therefore, This represents the total number of negative samples.

[0198] It should be noted that in practical applications, more advanced loss metrics can be used to replace the baseline loss for optimization.

[0199] When the i-th sample contains the j-th label ,otherwise , v i Let be the feature vector of the i-th sample. p j This is the class proxy vector corresponding to the j-th class. The cosine similarity loss represents the positive samples. The cosine similarity loss, representing the negative samples, is calculated using the following formula:

[0200] (2-2);

[0201] (2-3);

[0202] in, It is the interval factor.

[0203] b) Loss function for irrelevant sample pairs

[0204] To address the limitation of proxy loss in failing to constrain irrelevant samples with multiple labels, this embodiment introduces a small amount of crucial irrelevant sample pair information. The formula for calculating the irrelevant sample pair loss function is as follows:

[0205] (2-4);

[0206] Where D ’ M’ Let M' be the set of sample pairs containing multiple labels in the training set, and y be the number of such pairs. i Let |y| be the label of the i-th sample. i ·y j |=0 means that the i-th sample and the j-th sample have no common labels. When the i-th sample and the j-th sample have no common labels (that is, the i-th sample and the j-th sample are irrelevant sample pairs), then... =1, This represents the total number of irrelevant sample pairs.

[0207] c) Mixed loss function

[0208] To adjust the weights between the surrogate loss and the irrelevant sample pair loss, a balancing hyperparameter β is introduced in this embodiment. The formula for calculating the overall mixed loss function (2-5) is as follows:

[0209] (2-5);

[0210] Finally, the optimized network parameters θ are obtained using formula (2-6). * and class proxy vector set P * :

[0211] (2-6);

[0212] The training process of the feature extraction model provided in the embodiments of this application is described below.

[0213] During the training process of this model, the input information includes: training set D MHash bits K, training batch samples D batch Finally, the optimized network F is obtained. θ and class proxy vector set P C The training process includes the following steps:

[0214] Step S701, initialize network parameters θ=θ (0) Proxy vector set P C =P C (0) The number of iterations T=0;

[0215] Step S702, T ranges from 0 to 100, and the following iterative steps are performed to obtain the trained feature extraction model:

[0216] a), from D M Random sampling batch sample D batch ;

[0217] b) Calculate D batch The feature vector V of all samples i =F θ (T) (x i );

[0218] c) Calculate the agency loss L according to formula (2-1) proxy (D batch , P C (T) );

[0219] d) Calculate the irrelevant sample pair loss L according to formula (2-4). neg_pair (D batch ’ );

[0220] e) Calculate the total loss L according to formula (2-5) total (D batch , P C (T) );

[0221] f) Calculate the gradient of the total loss;

[0222] g) Update network parameters θ using backpropagation (T+1) Proxy vector set P C (T+1) ;

[0223] h)T = T + 1.

[0224] The method provided by the embodiments of this application can provide more accurate multi-label retrieval results and has good training time and stability during model training.

[0225] Table 1. Performance comparison of the retrieval method provided in this application and retrieval methods in related technologies on the average accuracy of the standard multi-label image retrieval datasets Flickr-25k and NUS-WIDE.

[0226]

[0227] Table 2. Performance comparison results of the retrieval methods provided in this application and related technologies on the standard multi-label image retrieval datasets VOC-2007 and VOC-2012, based on average accuracy.

[0228]

[0229] As can be seen from Tables 1 and 2, the method provided in this application embodiment outperforms existing methods under different datasets and different hash bit lengths, achieving optimal retrieval performance. Furthermore, the multi-label retrieval framework provided in this application embodiment can be applied not only to image retrieval, but also to any multi-label retrieval task utilizing deep learning in the embedding space, such as video retrieval, audio retrieval, and cross-modal retrieval.

[0230] It is understood that in the embodiments of this application, the content involving user information, such as data related to retrieval reference information, requires user permission or consent when the embodiments of this application are applied to specific products or technologies, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0231] The following description continues to illustrate the exemplary structure of the retrieval device 443 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software module stored in the retrieval device 443 of the memory 430 may include:

[0232] The first acquisition module 4431 is used to respond to the received retrieval request and acquire retrieval reference information and a trained feature extraction model. The loss function of the trained feature extraction model during the training process includes at least a first loss function component that can measure distance exclusion of multi-label irrelevant sample information.

[0233] The first processing module 4432 is used to extract features from the retrieval reference information using the trained feature extraction model to obtain a retrieval reference vector.

[0234] The first determining module 4433 is used to determine at least one target object from the retrieval database based on the retrieval reference vector;

[0235] The first output module 4434 is used to output the at least one target object.

[0236] In some embodiments, the device further includes:

[0237] The second acquisition module is used to acquire a preset feature extraction model and a training dataset, wherein the training dataset includes multiple training samples and label vectors of each training sample.

[0238] The third acquisition module is used to acquire the proxy vectors of each class and the number of samples N required for each training of the model, where N is an integer greater than 1;

[0239] The first prediction module is used to obtain N target training samples from the training dataset using the feature extraction model when it is determined that the training termination condition has not been met, and to extract features from each target training sample using the feature extraction model to obtain the predicted feature vector of each target training sample.

[0240] The fourth acquisition module is used to acquire the loss function of the feature extraction model. The loss function includes a first loss function component that can measure distance exclusion of multi-label irrelevant sample information and a second loss function component that is used to cluster training samples.

[0241] The first training module is used to train the feature extraction model using the loss function, the predicted feature vectors of each target training sample, the label vectors of each target training sample, and the class proxy vectors, until the training termination condition is met, and a trained feature extraction model is obtained.

[0242] In some embodiments, the first training module is further configured to:

[0243] Based on the label vectors of each target training sample, at least one multi-label irrelevant sample pair is determined;

[0244] The first loss value is determined based on the first loss function component of the loss function and the predicted feature vector of the target training sample in each multi-label irrelevant sample pair;

[0245] The second loss value is determined based on the second loss function component of the loss function, the predicted feature vector of each target training sample, the label vector of each target training sample, and each class proxy vector.

[0246] The total loss value is determined based on the first loss value and the second loss value;

[0247] The feature extraction model is trained by backpropagation based on the loss value in order to adjust the parameters of the feature extraction model.

[0248] In some embodiments, the first training module is further configured to:

[0249] Based on the label vectors of each target training sample, determine whether the i-th target training sample and the j-th target training sample have any identical label;

[0250] When it is determined that the i-th target training sample and the j-th target training sample do not have any of the same labels, the i-th target training sample and the j-th target training sample are determined as a multi-label irrelevant sample pair, where i=1,2,…N, j=1,2,…N, and i is not equal to j.

[0251] In some embodiments, the first training module is further configured to:

[0252] Obtain the total number of pairs of multi-label unrelated samples;

[0253] Based on the predicted feature vectors of the two target training samples in each multi-label irrelevant sample pair, the first negative sample loss value of each multi-label irrelevant sample pair is determined.

[0254] The first average negative sample loss value is determined based on the first negative sample loss value of each multi-label irrelevant sample pair and the total logarithm.

[0255] The first average negative sample loss value is determined as the first loss value.

[0256] In some embodiments, the first training module is further configured to:

[0257] Based on the label vectors of each target training sample and the class proxy vectors, determine the target positive training sample and the target negative training sample corresponding to each class proxy vector;

[0258] Determine the total number of positive samples of the target positive training samples corresponding to each class of proxy vectors and the total number of negative samples of the target negative training samples corresponding to each class of proxy vectors;

[0259] Based on the agent vectors of each class and the predicted feature vectors of the target positive training samples corresponding to the agent vectors of each class, the loss value of the first positive sample is determined.

[0260] Based on the proxy vectors of each class and the predicted feature vectors of the target negative training samples corresponding to each proxy vector, the loss value of the second negative sample is determined.

[0261] A first average positive sample loss value is determined based on the first positive sample loss value and the total number of positive samples, and a second average negative sample loss value is determined based on the second negative sample loss value and the total number of negative samples.

[0262] The sum of the first average positive sample loss value and the second average negative sample loss value is determined as the second loss value.

[0263] In some embodiments, the device further includes:

[0264] The fifth module is used to obtain the number of training iterations;

[0265] The second determining module is used to determine that the training termination condition has been met when the number of training iterations reaches a preset threshold; or,

[0266] The third determining module is used to determine that the training termination condition has been met when the total loss value is less than a preset loss threshold.

[0267] In some embodiments, the device further includes:

[0268] The sixth acquisition module is used to acquire test data, which includes multiple test samples and label information for each test sample;

[0269] The second prediction module is used to extract features from each test sample using the trained feature extraction model to obtain the predicted feature vector of each test sample.

[0270] The fourth determining module is used to determine the prediction information of each test sample based on the prediction feature vector of each test sample, and to determine the evaluation index value of the trained feature extraction model based on the prediction information of each test sample and the corresponding label vector.

[0271] The seventh acquisition module is used to determine that the evaluation index value is less than a preset index threshold, and then acquire multiple target training samples again.

[0272] The second training module is used to continue training the trained feature extraction model using the multiple target training samples until the evaluation index value of the retrained feature extraction model reaches the index threshold.

[0273] In some embodiments, the device further includes:

[0274] The eighth acquisition module is used to acquire object information of each search object in the search database;

[0275] The second processing module is used to extract features from the information of each object using the trained feature extraction model, and obtain the object feature vector corresponding to each retrieval object.

[0276] The binarization module is used to perform binarization processing on the object feature vectors corresponding to each search object to obtain the binarized feature vectors of each search object;

[0277] The storage module is used to store the binary feature vectors of each search object into the search database.

[0278] In some embodiments, the retrieval database stores binary feature vectors of each retrieval object, and the first determining module is further configured to:

[0279] The retrieval reference vector is binarized to obtain the binarized reference vector of the retrieval reference information;

[0280] Determine the distances between the binarized reference vector and the binarized feature vectors of each retrieved object;

[0281] The search objects that are less than the distance threshold are identified as the target objects.

[0282] It should be noted that the description of the retrieval device in this application is similar to the description of the method embodiments described above, and has similar beneficial effects. For technical details not disclosed in this device embodiment, please refer to the description of the method embodiments of this application for understanding.

[0283] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the retrieval method described above in this application.

[0284] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the retrieval method provided in this application. For example, ... Figure 3 , Figure 5 , Figure 6 The retrieval method is shown.

[0285] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0286] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0287] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0288] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0289] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A retrieval method, characterized in that, The method includes: In response to a received retrieval request, retrieval reference information and a trained feature extraction model are obtained. The loss function of the trained feature extraction model during the training process includes at least a first loss function component that can measure distance exclusion of multi-label irrelevant sample information and a second loss function component that is used to cluster training samples. The trained feature extraction model is used to extract features from the retrieval reference information to obtain a retrieval reference vector. At least one target object is determined from the retrieval database based on the retrieval reference vector; Output the at least one target object, wherein training the feature extraction model includes: The first loss value is determined based on the first loss function component and the predicted feature vector of the target training sample in each multi-label irrelevant sample pair. Determine the total number of positive samples and the total number of negative samples in the target positive training samples corresponding to each class of proxy vector; based on the predicted feature vectors of each class of proxy vector, the target positive training samples and the target negative training samples corresponding to each class of proxy vector, determine the first positive sample loss value and the second negative sample loss value; based on the first positive sample loss value and the total number of positive samples, determine the first average positive sample loss value, and based on the second negative sample loss value and the total number of negative samples, determine the second average negative sample loss value; The sum of the first average positive sample loss value and the second average negative sample loss value is determined as the second loss value; the total loss value is determined based on the first loss value and the second loss value; the feature extraction model is trained by backpropagation based on the total loss value to adjust the parameters of the feature extraction model.

2. The method according to claim 1, characterized in that, The method further includes: Based on the label vectors of each target training sample, determine whether the i-th target training sample and the j-th target training sample have any identical label; When it is determined that the i-th target training sample and the j-th target training sample do not have any of the same labels, the i-th target training sample and the j-th target training sample are determined as a multi-label irrelevant sample pair, where i=1,2,…N, j=1,2,…N, and i is not equal to j.

3. The method according to claim 1, characterized in that, The step of determining the first loss value based on the first loss function component and the predicted feature vector of the target training sample in each multi-label irrelevant sample pair includes: Obtain the total number of pairs of multi-label unrelated samples; Based on the predicted feature vectors of the two target training samples in each multi-label irrelevant sample pair, the first negative sample loss value of each multi-label irrelevant sample pair is determined. The first average negative sample loss value is determined based on the first negative sample loss value of each multi-label irrelevant sample pair and the total logarithm. The first average negative sample loss value is determined as the first loss value.

4. The method according to claim 1, characterized in that, The method further includes: Based on the label vectors of each target training sample and the class proxy vectors, the target positive training sample and target negative training sample corresponding to each class proxy vector are determined.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Get the number of training iterations; When the number of training iterations reaches a preset threshold, the training termination condition is determined to have been met; or, When the total loss value is less than the preset loss threshold, the training termination condition is determined to have been met.

6. The method according to claim 1, characterized in that, The method further includes: Acquire test data, which includes multiple test samples and label information for each test sample; The trained feature extraction model is used to extract features from each test sample to obtain the predicted feature vector of each test sample. The prediction information of each test sample is determined based on the predicted feature vector of each test sample, and the evaluation index value of the trained feature extraction model is determined based on the prediction information of each test sample and the corresponding label vector. Once the evaluation index value is determined to be less than a preset index threshold, multiple target training samples are obtained again. The trained feature extraction model is further trained using the multiple target training samples until the evaluation index value of the retrained feature extraction model reaches the index threshold.

7. The method according to claim 1, characterized in that, The method further includes: Obtain object information for each search object in the search database; The trained feature extraction model is used to extract features from the information of each object to obtain the object feature vector corresponding to each retrieved object. The object feature vectors corresponding to each search object are binarized to obtain the binarized feature vectors of each search object; The binary feature vectors of each search object are stored in the search database.

8. The method according to claim 1, characterized in that, The retrieval database stores binary feature vectors for each retrieval object. Determining at least one target object from the retrieval database based on the retrieval reference vector includes: The retrieval reference vector is binarized to obtain the binarized reference vector of the retrieval reference information; Determine the distances between the binarized reference vector and the binarized feature vectors of each retrieved object; The search objects that are less than the distance threshold are identified as the target objects.

9. A retrieval device, characterized in that, The device includes: The first acquisition module is used to acquire retrieval reference information and a trained feature extraction model in response to the received retrieval request. The loss function of the trained feature extraction model during the training process includes at least a first loss function component that can measure distance exclusion of multi-label irrelevant sample information and a second loss function component that is used to cluster training samples. The first processing module is used to extract features from the retrieval reference information using the trained feature extraction model to obtain a retrieval reference vector. The first determining module is used to determine at least one target object from the retrieval database based on the retrieval reference vector; A first output module is used to output the at least one target object; The first training module is used to train the feature extraction model in the following manner: The first loss value is determined based on the first loss function component and the predicted feature vector of the target training sample in each multi-label irrelevant sample pair. Determine the total number of positive samples and the total number of negative samples in the target positive training samples corresponding to each class of proxy vector; based on the predicted feature vectors of each class of proxy vector, the target positive training samples and the target negative training samples corresponding to each class of proxy vector, determine the first positive sample loss value and the second negative sample loss value; based on the first positive sample loss value and the total number of positive samples, determine the first average positive sample loss value, and based on the second negative sample loss value and the total number of negative samples, determine the second average negative sample loss value; The sum of the first average positive sample loss value and the second average negative sample loss value is determined as the second loss value; the total loss value is determined based on the first loss value and the second loss value; the feature extraction model is trained by backpropagation based on the total loss value to adjust the parameters of the feature extraction model.

10. The apparatus according to claim 9, characterized in that, The first training module is also used to determine, based on the label vectors of each target training sample, whether the i-th target training sample and the j-th target training sample have any identical label; When it is determined that the i-th target training sample and the j-th target training sample do not have any of the same labels, the i-th target training sample and the j-th target training sample are determined as a multi-label irrelevant sample pair, where i=1,2,…N, j=1,2,…N, and i is not equal to j.

11. The apparatus according to claim 9, characterized in that, The first training module is also used to obtain the total number of pairs of multi-label unrelated samples; Based on the predicted feature vectors of the two target training samples in each multi-label irrelevant sample pair, the first negative sample loss value of each multi-label irrelevant sample pair is determined. The first average negative sample loss value is determined based on the first negative sample loss value of each multi-label irrelevant sample pair and the total logarithm. The first average negative sample loss value is determined as the first loss value.

12. The apparatus according to claim 9, characterized in that, The first training module is further configured to determine the target positive training sample and the target negative training sample corresponding to each class proxy vector based on the label vector of each target training sample and the class proxy vector.

13. The apparatus according to any one of claims 9 to 12, wherein the apparatus further comprises: The fifth module is used to obtain the number of training iterations; The second determining module is used to determine that the training termination condition has been met when the number of training iterations reaches a preset threshold. or, The third determining module is used to determine that the training termination condition has been met when the total loss value is less than a preset loss threshold.

14. The apparatus of claim 9, further comprising: The sixth acquisition module is used to acquire test data, which includes multiple test samples and label information for each test sample; The second testing module is used to extract features from each test sample using the trained feature extraction model to obtain the predicted feature vector of each test sample. The fourth determining module is used to determine the prediction information of each test sample based on the prediction feature vector of each test sample, and to determine the evaluation index value of the trained feature extraction model based on the prediction information of each test sample and the corresponding label vector. The seventh acquisition module is used to determine that the evaluation index value is less than a preset index threshold, and then acquire multiple target training samples again. The second training module is used to continue training the trained feature extraction model using the multiple target training samples until the evaluation index value of the retrained feature extraction model reaches the index threshold.

15. The apparatus of claim 9, further comprising: The eighth acquisition module is used to acquire object information of each search object in the search database; The second processing module is used to extract features from the information of each object using the trained feature extraction model, and obtain the object feature vector corresponding to each retrieval object. The binarization module is used to perform binarization processing on the object feature vectors corresponding to each search object to obtain the binarized feature vectors of each search object; The storage module is used to store the binary feature vectors of each search object into the search database.

16. The apparatus according to claim 15, characterized in that, The first determining module is further configured to perform binarization processing on the retrieval reference vector to obtain a binarized reference vector of the retrieval reference information; Determine the distances between the binarized reference vector and the binarized feature vectors of each retrieved object; The search objects that are less than the distance threshold are identified as the target objects.

17. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the method according to any one of claims 1 to 8.

18. A computer-readable storage medium storing executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the method described in any one of claims 1 to 8.

19. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the method described in any one of claims 1 to 8.

Images