Image retrieval model training and retrieval method, device, equipment and medium

Abstract

The present application discloses a training and retrieval method, device, equipment and medium for an image retrieval model, which belongs to the field of artificial intelligence. The image retrieval model includes an embedded vector network and a quantized index network, and the training method of the image retrieval model includes: obtaining n sample triplets for training the image retrieval model; inputting the basic feature vectors of the n sample triplets into the embedded vector network; screening out a first sample triplet set for training the quantized index network according to the error of the feature vector output by the embedded vector network; inputting the basic feature vectors of the n sample triplets into the quantized index network; screening out a second sample triplet set for training the embedded vector network according to the error of the quantized index output by the quantized index network; training the quantized index network based on the first sample triplet set, and training the embedded vector network based on the second sample triplet set. The above method improves the accuracy of image retrieval.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method, device, equipment and medium for training and retrieving an image retrieval model. Background Art

[0002] When performing image retrieval based on a query image, multiple candidate images are often recalled and ranked to select candidate images with higher confidence as retrieval results.

[0003] In the related art, the feature vector of the query image is often first obtained through an embedded vector network, the feature vector is PQ quantized (Product Quantization) to obtain a quantization index, and then m feature vectors matching the quantization index are found from the quantization codebook, and m candidate images corresponding to the m feature vectors are recalled. Then, the distance between the feature vectors of the m candidate images and the feature vector of the query image is used to sort the results, and the candidate image with a higher ranking is selected as the final recalled image.

[0004] In the related art, PQ quantization divides the value of each dimension of the feature vector into multiple segments, each segment is represented by a different digital number (for example, if a certain bit value is a floating point number between 0 and 1, it may be divided into 10 segments of 0.1, ... 0.9, 1.0, and each segment is represented by a number from 1 to 10). During retrieval, candidate images quantized to the same segment are recalled. However, the PQ quantization method easily causes similar features to be split into two adjacent segments, and critical samples are prone to missed recall or multiple recall (for example, the critical samples of segment 1 and segment 2 may be similar to segment 1 or segment 2, and recalling any segment alone will result in missed recall, and recalling two segments will increase false recalls). Summary of the invention

[0005] The present application provides a training and retrieval method, device, equipment and medium for an image retrieval model, which can improve the accuracy of image retrieval. The technical solution is as follows:

[0006] According to one aspect of the present application, a training method for an image retrieval model is provided, wherein the image retrieval model includes an embedding vector network and a quantization index network, wherein the embedding vector network is used to obtain a feature vector of an image, and the quantization index network is used to extract a quantization index of the image; the method includes:

[0007] Obtain n sample triplets for training an image retrieval model; the sample triplets include a target sample, a positive sample that forms a similar sample pair with the target sample, and a negative sample that does not form a similar sample pair with the target sample; n is a positive integer greater than 1;

[0008] Inputting the basic feature vectors of the n sample triplets into the embedding vector network; selecting the first sample triplet set for training the quantization index network according to the error of the feature vector output by the embedding vector network;

[0009] Inputting the basic feature vectors of the n sample triplets into the quantization index network; selecting a second sample triplet set for training the embedding vector network according to the error of the quantization index output by the quantization index network;

[0010] A quantization index network is trained based on the first set of sample triples, and an embedding vector network is trained based on the second set of sample triples.

[0011] According to another aspect of the present application, there is provided an image retrieval method, the method comprising:

[0012] Get the basic feature vector of the query image;

[0013] Input the basic feature vector into the quantized index network and the embedding vector network;

[0014] Obtaining a quantized index of the query image through a quantized index network, and obtaining a feature vector of the query image through an embedding vector network;

[0015] Based on the quantization index, the feature vectors of m candidate images are obtained by indexing from the quantization codebook; the quantization codebook stores the mapping relationship between the quantization index and the feature vectors of the m candidate images; m is a positive integer;

[0016] Calculate the fifth distances between the feature vectors of the m candidate images and the feature vector of the query image respectively, and obtain m fifth distances;

[0017] Among the m fifth distances sorted from small to large, the candidate images corresponding to the fifth distances ranked in the top z% are screened out.

[0018] According to another aspect of the present application, a training device for an image retrieval model is provided, wherein the image retrieval model includes an embedding vector network and a quantization index network, wherein the embedding vector network is used to obtain a feature vector of an image, and the quantization index network is used to extract a quantization index of the image; the device includes:

[0019] An acquisition module is used to acquire n sample triplets for training an image retrieval model; the sample triplets include a target sample, a positive sample that forms a similar sample pair with the target sample, and a negative sample that does not form a similar sample pair with the target sample;

[0020] A screening module, used to input the basic feature vectors of the n sample triplets into the embedding vector network; and screen out a first sample triplet set for training the quantized index network according to the feature vectors output by the embedding vector network;

[0021] The screening module is further used to input the basic feature vectors of the n sample triplets into the quantization index network; and screen out a second sample triplet set for training the embedding vector network according to the quantization index output by the quantization index network;

[0022] A training module is used to train a quantization index network based on the first sample triple set and to train an embedding vector network based on the second sample triple set.

[0023] According to another aspect of the present application, an image retrieval device is provided, the device comprising:

[0024] An acquisition module, used to acquire a basic feature vector of a query image;

[0025] An input module, used for inputting the basic feature vector into the quantization index network and the embedding vector network;

[0026] The acquisition module is further used to acquire the quantized index of the query image through the quantized index network, and acquire the feature vector of the query image through the embedding vector network;

[0027] An indexing module, used to obtain feature vectors of m candidate images from a quantization codebook based on a quantization index; the quantization codebook stores a mapping relationship between the quantization index and the feature vectors of the m candidate images;

[0028] A calculation module, used for respectively calculating the fifth distances between the feature vectors of the m candidate images and the feature vector of the query image, to obtain m fifth distances;

[0029] The screening module is used to screen out candidate images corresponding to the fifth distances ranked in the top z% from the m fifth distances sorted from small to large.

[0030] According to one aspect of the present application, a computer device is provided, comprising: a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the training method of the image retrieval model as described above, and / or the image retrieval method.

[0031] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the training method of the image retrieval model as described above, and / or the image retrieval method.

[0032] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned image retrieval model training method and / or image retrieval method.

[0033] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0034] By training the embedding vector network and the quantization index network with sample triplets, not only the first set of sample triplets screened by the embedding vector network is used as the sample triplets for training the quantization index network, but also the second set of sample triplets screened by the quantization index network is used as the sample triplets for training the embedding vector network. The above method enables the embedding vector network to support the calculation of the distance between the feature vectors of the m candidate images and the feature vector of the query image, and also makes the quantization index of the query image obtained by the quantization index network more accurate. The training method of the above image retrieval model eliminates noise sample triplets, and at the same time makes the prediction effects of the quantization index network and the embedding vector network on positive and negative samples similar. The noise sample triplets are predicted by dual branches, and the dual branches learn from each other's excellent sample triplets to achieve denoising learning, so that the dual branches have similar prediction effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 is a schematic diagram of PQ quantization provided by an exemplary embodiment of the present application;

[0037] Figure 2 is a schematic diagram of an implementation environment of an image retrieval model of an exemplary embodiment of the present application;

[0038] Figure 3 is a flowchart of a method for training an image retrieval model of an exemplary embodiment of the present application;

[0039] Figure 4 is a schematic diagram of a training system for an image retrieval model of an exemplary embodiment of the present application;

[0040] Figure 5is a flowchart of a method for training an image retrieval model of another exemplary embodiment of the present application;

[0041] Figure 6 is a flowchart of a method for training an image retrieval model of another exemplary embodiment of the present application;

[0042] Figure 7 is a flowchart of a method for training an image retrieval model of another exemplary embodiment of the present application;

[0043] Figure 8 is a schematic diagram of a training system for an image retrieval model of another exemplary embodiment of the present application;

[0044] Fig. 9 is a flowchart of a method for training an image retrieval model of another exemplary embodiment of the present application;

[0045] Fig.10 is a flowchart of an image retrieval method provided by an exemplary embodiment of the present application;

[0046] Fig.11 is a schematic diagram of a system for using an image retrieval model provided by an exemplary embodiment of the present application;

[0047] Fig.12 is a structural block diagram of a training device for an image retrieval model provided by an exemplary embodiment of the present application;

[0048] Fig.13 is a structural block diagram of an image retrieval device provided by an exemplary embodiment of the present application;

[0049] Fig.14 is a structural block diagram of a computer device provided by an exemplary embodiment of the present application;

[0050] Fig.15 is a schematic diagram of a data sharing system provided by an exemplary embodiment of the present application;

[0051] Fig.16 is a schematic diagram of a blockchain structure provided by an exemplary embodiment of the present application;

[0052] Fig.17 It is a schematic diagram of a new block generation process provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0054] First, the nouns involved in the embodiments of the present application are briefly introduced:

[0055] PQ quantization: During the training phase, refer to Figure 1 , for N training samples, assuming that the sample dimension is 128 dimensions, it is divided into 4 subspaces, then the dimension of each subspace is 32 dimensions, and in each subspace, the feature vectors are clustered using k-means (a clustering algorithm) ( Figure 1 256 cluster centers are shown in the figure), and a codebook can be obtained for each subspace. And each subsegment of each training sample can be approximated by the cluster center of the subspace, and the corresponding code is the value of the cluster center. The N training samples finally obtained constitute an index codebook. For the samples to be quantized, they are divided in the same way, and then the cluster centers closest to them are found one by one in each subspace, and then each subsegment is represented by the value of the cluster center, which completes the index vector of the sample to be quantized.

[0056] In the retrieval stage, the feature vector of the query image is divided into 4 sub-segments, and then in each subspace, the distance from the sub-segment to all cluster centers in the subspace is calculated to obtain 4*256 distances, which are used as the distance table. When calculating the distance from a sample to the query vector, for example, when calculating the distance from the sample encoded as (124, 56, 132, 222) to the query vector, we can take the distances corresponding to each sub-segment from the distance table. For example, for the sub-segment encoded as 124, we can take out the distance numbered 124 from the first 256 distances calculated. After taking out the distances corresponding to all sub-segments, we sum up the distances of these sub-segments to obtain the asymmetric distance between the sample and the query sample. After all distances are calculated, recall the samples with the highest distance ranking.

[0057] In this application, considering that the PQ quantization method has a slow calculation speed and easily causes similar basic features to be split into two adjacent cluster centers, a method of constructing a deep symbolic quantization network is adopted.

[0058] Deep symbol quantization network (quantization index network): First, for the D-dimensional feature vectors of N training samples, the value range of each dimension after vector normalization is a floating point number from -1 to 1, and the D-dimensional feature vector is compressed to a binary code with a specified number of bits and values ​​of 0 and 1, which is symbol quantization. For example, after normalization of the 4-dimensional feature vector, (-1, 1, 0.5, -0.2) is obtained, and after symbol quantization, the quantization index is (0, 1, 1, 0).

[0059] Computer Vision (CV): Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further process images so that the computer processes them into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0060] Artificial Intelligence Cloud Service: The so-called artificial intelligence cloud service is generally also referred to as AIaaS (AI as aService, Chinese for "AI as a Service"). This is a mainstream service mode of artificial intelligence platform. Specifically, the AIaaS platform will split several common AI (Artificial Intelligence) services and provide independent or packaged services in the cloud. This service model is similar to opening an AI theme mall: all developers can access and use one or more artificial intelligence services provided by the platform through the API (Application Program Interface) interface. Some senior developers can also use the AI ​​framework and AI infrastructure provided by the platform to deploy and operate their own cloud artificial intelligence services.

[0061] The solution of the embodiment of the present application includes an image retrieval model training phase and an image retrieval phase. Figure 2 The computer system 200 in which the training method and the use method of the image retrieval model provided by an exemplary embodiment of the present application are located is shown. Among them, the terminal 220 is used to train the image retrieval model and / or use the image retrieval model, and the server 240 is used to train the image retrieval model and / or use the image retrieval model.

[0062] The terminal 220 is installed and runs a client that supports image retrieval. The client can be any one of an application, web page, and applet that supports image retrieval. The terminal 220 can train an image retrieval model and / or use an image retrieval model. Optionally, the client installed on the terminal 220 is a client on an operating system platform (Android or IOS). The terminal 220 can generally refer to one of a plurality of terminals, and this embodiment is only illustrated by the terminal 220. The device types of the terminal 220 include: at least one of a smart phone, a smart watch, a smart TV, a tablet computer, an e-book reader, an MP3 player, an MP4 player, a laptop portable computer, and a desktop computer. Those skilled in the art can know that the number of the above-mentioned terminals can be more or less. For example, the above-mentioned terminal can be only one, or the above-mentioned terminal can be dozens or hundreds, or more. The embodiment of the present application does not limit the number and device type of the terminal.

[0063] The terminal 220 is connected to the server 240 via a wireless network or a wired network.

[0064] The server 240 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Exemplarily, the server 240 includes a processor 244 and a memory 242, and the memory 242 includes a receiving module 2421, a control module 2422, and a sending module 2423. The receiving module 2421 is used to receive a request sent by the client, such as training an image retrieval model and using an image retrieval model; the control module 2422 is used to control the training and use of the image retrieval model; the sending module 2423 is used to send a response to the terminal, such as returning the retrieved image to the client, or returning the trained image retrieval model. The server 240 is used to provide background services for image retrieval and / or image model training. Optionally, the server 240 undertakes the main computing work, and the terminal 220 undertakes the secondary computing work; or, the server 240 undertakes the secondary computing work, and the terminal 220 and the second terminal 160 undertake the main computing work; or, the server 240 and the terminal 220 adopt a distributed computing architecture for collaborative computing.

[0065] The following example illustrates that the training method of the image retrieval model and the image retrieval method can be executed by a computer device. Optionally, the computer device can be a terminal or a server. The present application does not limit the execution subject of the training method of the image retrieval model and the image retrieval method.

[0066] In order to achieve image retrieval by building a deep quantization network, Figure 3 is a flowchart of a method for training an image retrieval model according to an exemplary embodiment of the present application. Figure 2The computer device shown in the figure is used to perform an example, wherein the image retrieval model includes an embedding vector network and a quantization index network, the embedding vector network is used to obtain a feature vector of the image, and the quantization index network is used to extract a quantization index of the image, and the method includes:

[0067] Step 320, obtaining n sample triplets for training the image retrieval model;

[0068] The sample triplet includes a target sample, a positive sample that forms a similar sample pair with the target sample, and a negative sample that does not form a similar sample pair with the target sample, and n is a positive integer greater than 1.

[0069] Image retrieval: Get the quantization index by querying the feature vector of the image, then find m feature vectors matching the quantization index from the quantization codebook, calculate the distance between the m feature vectors and the feature vector of the query image, and select the image corresponding to the top-ranked distance as the final screened image, where m is a positive integer.

[0070] Quantitative index: When the image retrieval model performs well, one quantitative index often corresponds to one type of image. For example, all sample images are divided into two categories: human category and animal category. There is a quantitative index corresponding to the human category and another quantitative index corresponding to the animal category.

[0071] It should be noted that in order to better train the quantitative index network and make the image retrieval model perform well in the distance calculation of the quantitative index and feature vector as much as possible, sample triplets are often used to train the image retrieval model.

[0072] Step 340, inputting the basic feature vectors of the n sample triplets into the embedding vector network; and selecting a first set of sample triplets for training the quantization index network according to the error of the feature vector output by the embedding vector network;

[0073] Basic feature vector: refers to the vector of basic image features in the sample triplet. The basic image features include but are not limited to color features, texture features, shape features and spatial relationship features.

[0074] The embedding vector network generates n groups of triple feature vectors from the basic feature vectors of n sample triplets, and screens out a first set of sample triplets for training the quantized index network based on the n groups of triple feature vectors.

[0075] Step 360, inputting the basic feature vectors of the n sample triplets into the quantization index network; and selecting a second set of sample triplets for training the embedding vector network according to the error of the quantization index output by the quantization index network;

[0076] The quantized index network generates n sets of triplet quantized indexes from the basic feature vectors of the n sample triplets, and selects a second set of sample triplets for training the embedding vector network.

[0077] Step 381, training a quantized index network based on the first sample triplet set;

[0078] Step 382: train an embedding vector network based on the second set of sample triples.

[0079] A quantization index network is trained based on the first set of sample triples, and an embedding vector network is trained based on the second set of sample triples.

[0080] In summary, by training the embedding vector network and the quantization index network through sample triplets, not only the first set of sample triples screened by the embedding vector network is used as the sample triplets for training the quantization index network, but also the second set of sample triples screened by the quantization index network is used as the sample triplets for training the embedding vector network. The above method enables the embedding vector network to support the calculation of the distance between the feature vectors of the m candidate images and the feature vector of the query image, and also makes the quantization index of the query image obtained by the quantization index network more accurate.

[0081] Figure 4 It is a training system for an image retrieval model provided by an exemplary embodiment of the present application, wherein the training system for the image retrieval model includes an embedding vector network 401 and a quantization index network 402.

[0082] The embedding vector network 401 is used to generate n groups of triple feature vectors from the basic feature vectors of n sample triplets, and the triple feature vectors can be used to calculate the similarity distance between the target sample, the positive sample and the negative sample in the sample triplet.

[0083] The quantization index network 402 is used to generate n groups of triplet quantization indexes from the basic feature vectors of the n sample triplets, and the triplet quantization indexes are used to quantize the sample triplets.

[0084] Figure 4 The training system of the image retrieval model shown in the figure constructs sample triplets from samples, and inputs the constructed n sample triplets into the embedding vector network 401 and the quantization index network 402 respectively, and then screens the sample triplets. The loss function of the quantization index network is calculated by the first set of sample triplets screened by the embedding vector network 401, and the quantization loss network 402 is trained based on the loss function value; the loss function of the embedding vector network is calculated by the second set of sample triplets screened by the quantization index network 402, and the embedding vector network 401 is trained based on the loss function value.

[0085] It is worth noting that the above screening of n sample triplets through two branches is based on the principle that the performance of noise sample triplets (sample triplets in which negative samples and target samples constitute positive sample pairs) in the embedding vector network 401 and the quantization index network 402 is not necessarily consistent, such as: the negative samples and target samples in the noise sample triplets are considered to constitute positive sample pairs through the embedding vector network 401, that is, a high loss function value is generated, and the negative samples and target samples in the noise sample triplets are considered to constitute negative sample pairs through the quantization index network 402, resulting in a low loss function value. At this time, if the noise sample triplets with inconsistent performance are removed, the embedding vector network 401 and the quantization index network 402 can be better trained.

[0086] For step 320, n sample triplets are obtained for training the image retrieval model.

[0087] In the image retrieval model of the present application, sample triplets need to be introduced to train the embedding vector network and the quantization index network, and the sample triplets are constructed by several similar sample pairs.

[0088] Similar sample pairs: In the image retrieval model of the present application, similar sample pairs need to be introduced to train the embedding vector network and the quantization index network. The similar sample pairs include the target image and positive samples that are extremely similar or identical to the target image. Optionally, two adjacent frames of images in the video can construct similar sample pairs.

[0089] Sample triples: In the image retrieval model of the present application, in order to generate n sample triplets, in each batch of R similar sample pairs, for the target sample in each similar sample pair, randomly select an image from each of the remaining (R-1) similar sample pairs, calculate the distance between the feature vector of the (R-1) image and the feature vector of the target sample, sort the (R-1) distances from small to large, select the image corresponding to the first n distances and the similar sample pair where the target sample is located to form n sample triplets. That is, obtain n sample triplets for training the image retrieval model.

[0090] Schematically, the method of constructing a sample triple is as follows:

[0091] The 26 similar sample pairs include (A, A'), (B, B'), (C, C'), (D, D'), ..., (Z, Z'). For the similar sample pair (A, A'), A is the target sample (anchor), and A' is the positive sample (positive). From each of the remaining similar sample pairs (B, B'), (C, C'), (D, D'), ..., (Z, Z'), one image is randomly selected, such as B, C', D, E' ..., Z. The distances between the feature vectors of B, C', D, E' ..., Z and the feature vector of the target sample A are calculated respectively to obtain 25 distances. The 25 distances are sorted from small to large, and the images corresponding to the first 20 distances are selected to form 20 sample triplets with the similar sample pair (A, A') where the target sample A is located. That is, 20 sample triplets for training the image retrieval model are obtained.

[0092] In summary, the above method realizes the construction of n sample triplets from similar sample pairs, so that the obtained n sample triplets can be used for the training of the image retrieval model provided in this application.

[0093] Figure 5 A training method for an image retrieval model provided by an exemplary embodiment of the present application is shown, and the method is applied to Figure 2 The computer device shown in FIG. 1 is used as an example to illustrate, wherein steps 320, 360, 381 and 382 are all the same as Figure 3 The same as shown.

[0094] For step 340, the basic feature vectors of the n sample triplets are input into the embedding vector network; and according to the error of the feature vector output by the embedding vector network, a first set of sample triplets for training the quantization index network is screened out.

[0095] Step 340 may include the following steps:

[0096] Step 341, obtaining n groups of triplet feature vectors output by the embedding vector network for n sample triples;

[0097] By inputting n sample triplets into the embedding vector network 401, the embedding vector network 401 outputs n groups of triplet feature vectors.

[0098] Step 342, calculating n first error losses corresponding to n groups of triplet feature vectors;

[0099] Calculating n first error losses corresponding to n groups of triplet feature vectors includes:

[0100] For each group of triple feature vectors, a first distance between the feature vector of the target sample and the feature vector of the positive sample is calculated; for each group of triple feature vectors, a second distance between the feature vector of the target sample and the feature vector of the negative sample is calculated; and a first error loss between a difference between the first distance and the second distance and a first distance threshold is calculated, where the first distance threshold is a threshold of a difference between a distance between the target sample and the positive sample and a distance between the target sample and the negative sample.

[0101] Schematically, in each set of triples (a, p, n), the triple feature vector is L2 normalized, and then the first error loss is calculated. a (anchor) represents the target sample, p (positive) represents the positive sample, and n (nagative) represents the negative sample.

[0102] The first error loss is calculated as follows:

[0103] l tri =max(||X a -X p ||-||X a -X n ||+α, 0); (1)

[0104] Among them, l tri represents the first error loss, X a Represents the feature vector of the target sample, X p Represents the feature vector of the positive sample, X n represents the feature vector of the negative sample, α represents the first distance threshold, ||X a -X p || represents the first distance, ||X a -X n || represents the second distance, which is optional. The α value is 4, which means the distance between the feature vector of the target sample and the feature vector of the negative sample is greater than 4. The purpose of using L2 normalization is to make the feature space of the triplet sample in the range of 0 to 1, avoiding the feature space being too large, which is not conducive to optimizing the training image retrieval model.

[0105] Step 343 , from the results of sorting the n first error losses from small to large, select the sample triplets corresponding to the n1 first error losses sorted within the first selected range, and add them to the first sample triplet set for training the quantization index network.

[0106] Through step 342, n first error losses can be obtained, and the n first error losses are sorted from small to large, and the sample triplets corresponding to the n1 first error losses sorted within the first selected range are screened out and added to the first sample triplet set used to train the quantization index network, where n1 is a positive integer less than n.

[0107] Optionally, among the results of sorting the n first error losses from small to large, according to a preset x value, sample triplets corresponding to the n1 first error losses ranked in the top x% are screened out and added to the first sample triplet set for training the quantization index network. Schematically, the x value is set based on the ratio of noise sample triplets to the n sample triplets, and optionally, the ratio is obtained based on prediction or calibration. Schematically, the x value is slightly larger than the ratio of noise sample triplets to the n sample triplets. x is a positive number.

[0108] Illustratively, the sorting results of n first error losses constitute Lem_list, and n1 sample triplets corresponding to n1 first error losses ranked in the top 85% in Lem_list are selected as the first sample triplet set.

[0109] It is worth mentioning that when training the image retrieval model, according to the results of multiple experiments, it can be predicted that there are 10% noise sample triplets in the n sample triplets (that is, for a similar sample pair in the n sample triplets, there may be 10% of the images in other similar sample pairs that can form a positive sample pair with the target sample), and the n1 sample triplets corresponding to the first 85% of the n1 first error losses in Lem_list are selected as the first sample triplet set. However, in each round (epoch) of image retrieval model training, it cannot always be guaranteed that the noise sample triplets are ranked in the last 10%, especially in the first few rounds of training, so taking 5% less can avoid taking noise sample triplets to a certain extent.

[0110] In summary, the above method realizes the calculation of the loss function value of n groups of triplet feature vectors of n sample triplets by designing the loss function of the embedded vector network, and screens the first set of sample triplets for training the quantization index network based on the loss function value, and further uses cleaner sample triplets to optimize the quantization effect of the quantization index network.

[0111] Figure 6 A training method for an image retrieval model provided by an exemplary embodiment of the present application is shown, and the method is applied to Figure 2 The computer device shown in FIG. 1 is used as an example to illustrate, wherein steps 320, 340, 381 and 382 are all the same as Figure 3 The same as shown.

[0112] For step 360, the basic feature vectors of the n sample triplets are input into the quantization index network; according to the error of the quantization index output by the quantization index network, a second set of sample triplets for training the embedding vector network is screened out;

[0113] Step 360 may include the following steps:

[0114] Step 361, obtaining n groups of triplet quantization indexes output by the quantization index network for n sample triples;

[0115] By inputting the basic feature vectors of n sample triplets into the quantization index network, n sets of triplet quantization indexes are output.

[0116] Step 362, calculating n second error losses corresponding to n groups of triplet quantization indices;

[0117] The method of calculating n second error losses corresponding to n groups of triplet quantization indices includes:

[0118] For each group of triple quantization indexes, a first triple loss of the triple quantization indexes is calculated; for each group of triple quantization indexes, a first quantization error loss of the triple quantization indexes is calculated; and a weighted sum of the first triple loss and the first quantization error loss is performed to obtain a second error loss.

[0119] Calculate the first triplet loss. Schematically, activate the triplet quantization index through the activation function, and then calculate the first triplet loss of the quantization index. To ensure that the sample triplet is distinguishable in the quantization space, the distance between the target sample and the negative sample of the sample triplet needs to be large enough. Optionally, set margin (α in formula (1)) to 160.

[0120] The calculation method of the first triplet loss is similar to the calculation method of the first error loss mentioned above, and will not be repeated here. The difference here is that the dimension of the triplet quantization index is inconsistent with the dimension of the triplet feature vector.

[0121] The quantization index network 402 outputs n groups of triple quantization indexes. Optionally, the triple quantization index is 256-dimensional. Each dimension is converted into a (-1, 1) value through an activation function, and then symbol quantization is performed to finally obtain a 256-dimensional triple quantization index.

[0122] Calculate the first quantization error loss. To better train the quantization index network, the value of each dimension of the triplet quantization index to be used should be as close to -1 or 1 as possible (if the value of each dimension is near the critical value, that is, 0, it is easy to cause similar basic feature vectors to be quantized to different segments). Therefore, the quantization loss function is set as follows (2) (3). First, calculate:

[0123]

[0124] where u i The value of each dimension of the triplet quantization index without sign quantization (values ​​less than 0 are quantized to 0, and values ​​greater than 0 are quantized to 1), u i The range of b is (-1, 1), i The value of each dimension of the quantized index for the transformed triplet (-1 or 1);

[0125] Then calculate b i and u i The regression loss between

[0126]

[0127] Among them, L coding is the first quantization error loss.

[0128] The activation function can be set to Tanh function or sgn function. It is worth noting that the sgn function is not differentiable at position 0 (+0 and -0), that is, it is impossible to calculate the gradient (and thus cannot be used in deep learning based on sgd gradient feedback), while the Tanh activation is differentiable and can be mapped to between -1 and 1. In addition, the activation function can also use sigmoid (activated to between 0 and 1), and then use 0 and 1 as quantization targets (rather than -1 and 1). Among them, tanh can obtain (-1, 1) faster, so the training effect is better.

[0129] The second error loss is calculated. Based on the first triplet loss and the first quantization error loss obtained above, a weighted sum is performed to obtain the second error loss.

[0130] L q =w 21 L triplet +w 22 L coding ; (4)

[0131] Among them, at this time L q The output is the second error loss, L triplet is the first triplet loss, L coding is the first quantization error loss, w 21 and w 22 is the weight, which is optional. Since the first quantization error loss converges faster than the first triplet loss, in order to ensure that the first triplet loss is dominant in the overall second error loss, so as to ensure that the embedding vector network always has the ability to measure similarity, here w 21 Set to 1, w 22 Set to 0.5.

[0132] It is worth noting that the above21 and w 22 The value of is not fixed, just ensure that w 22 The value is less than w 21 That's it.

[0133] Step 363, from the results of sorting the n second error losses from small to large, select the sample triples corresponding to the n2 second error losses sorted within the second selected range, and add them to the second sample triplet set used to train the embedding vector network.

[0134] Through step 362, n second error losses can be obtained, and the n second error losses are sorted from small to large, and the sample triplets corresponding to the n2 second error losses sorted within the second selected range are screened out and added to the second sample triplet set used to train the quantization index network.

[0135] Optionally, among the results of sorting the n second error losses from small to large, according to a preset y value, the sample triplets corresponding to the n2 second error losses ranked in the top y% are screened out and added to the second sample triplet set for training the quantized index network. Schematically, the y value is set based on the ratio of noise sample triplets to the n sample triplets, and optionally, the ratio is obtained based on prediction or calibration. Schematically, the y value is slightly larger than the ratio of noise sample triplets to the n sample triplets. y is a positive number.

[0136] Illustratively, the sorting results of n second error losses constitute Lq_list, and n2 sample triplets corresponding to the top 85% of n2 second error losses in Lq_list are selected as the second sample triplet set, where n2 is a positive integer less than n.

[0137] It is worth noting that when training the image retrieval model, according to the results of multiple experiments, it can be predicted that there are 10% noise sample triplets in the n sample triplets (that is, for a similar sample pair in the n sample triplets, there may be 10% of the images in other similar sample pairs that can form a positive sample pair with the target sample), and the n2 sample triplets corresponding to the n2 second error losses in the top 85% of Lq_list are selected as the second sample triplet set. However, in each round (epoch) of image retrieval model training, it cannot always be guaranteed that the noise sample triplets are ranked in the last 10%, especially in the first few rounds of training, so taking 5% less can avoid taking noise sample triplets to a certain extent.

[0138] In summary, the above method realizes the calculation of the loss function values ​​of n groups of triplet quantization indices of n sample triplets by designing the loss function of the quantization index network, and screens the second set of sample triplets for training the embedding vector network based on the loss function value, and further uses cleaner sample triplets to optimize the calculation effect of the feature vector of the embedding vector network.

[0139] Figure 7 A training method for an image retrieval model provided by an exemplary embodiment of the present application is shown, and the method is applied to Figure 2 The computer device shown in FIG. 1 is used as an example to illustrate, wherein steps 320, 340 and 360 are all the same as those in FIG. Figure 3 The same as shown.

[0140] For step 381, training a quantized index network based on the first sample triplet set includes:

[0141] Step 381-1, inputting the first sample triplet set into the quantization index network to obtain n1 third error losses;

[0142] Among them, the first sample triple set includes n1 sample triplets, and the first sample triplet is obtained based on the first error loss sorting of n groups of triple feature vectors output by the embedding vector network 401. For details, please refer to the above "for step 340".

[0143] The first sample triplet set is input into the quantization index network to obtain n1 third error losses including:

[0144] For each sample triplet of the first sample triplet set, a second triplet loss of the triplet feature vector is calculated through a quantization index network, wherein the triplet feature vector is a feature vector output by the embedding vector network; for each sample triplet of the first sample triplet set, a second quantization error loss of the triplet feature vector is calculated through a quantization index network; and a weighted sum of the second triplet loss and the second quantization error loss is performed to obtain a third error loss.

[0145] The method of calculating the n1 third error losses is similar to the method of calculating the n second error losses in the above step 362, and will not be described in detail.

[0146] Step 381 - 2 , training the quantization index network based on n1 third error losses.

[0147] The computer device trains a quantized index network based on n1 third error losses.

[0148] For step 382, ​​training an embedding vector network based on the second set of sample triples includes:

[0149] Step 382-1, inputting the second sample triplet set into the embedding vector network to obtain n2 fourth error losses;

[0150] The second sample triplet set includes n2 sample triplets, and the second sample triplet is obtained based on the second error loss sorting of the n groups of triplet quantization indices output by the quantization index network 402. For details, refer to the above “for step 360”.

[0151] Among them, the second sample triplet set is input into the embedding vector network to obtain n2 fourth error losses including:

[0152] For each sample triplet of the second sample triplet set, calculating a third distance between a feature vector of the target sample and a feature vector of the positive sample by using an embedding vector network;

[0153] For each sample triplet of the second sample triplet set, calculating a fourth distance between a feature vector of the target sample and a feature vector of the negative sample by using an embedding vector network;

[0154] A fourth error loss between a difference between the third distance and the fourth distance and a second distance threshold is calculated, where the second distance threshold is a threshold of a difference between a distance between the target sample and the positive sample and a distance between the target sample and the negative sample.

[0155] The method of calculating the n2 fourth error losses is similar to the method of calculating the n first error losses in the above step 342, and will not be repeated herein.

[0156] Step 382-2, training the embedding vector network based on n2 fourth error losses.

[0157] The computer device trains the embedding vector network based on the n2 fourth error loss.

[0158] In summary, the first sample triplet set is used to train the quantization index network, and the second sample triplet set is used to train the embedding vector network, which optimizes the quantization effect of the quantization index network and the feature vector calculation effect of the embedding vector network.

[0159] Figure 8 A training system for an image retrieval model provided by an exemplary embodiment of the present application is shown. Figure 4 The training system of the image retrieval model shown in the figure further adds a basic feature network 403.

[0160] based on Figure 3 The training method of the image retrieval model shown in the figure further includes, between step 320 and step 340: obtaining basic feature vectors of n sample triplets through a basic feature network.

[0161] The basic feature network 403 is used to extract basic image features of the input sample triplet, and the basic features include but are not limited to color features, texture features, shape features and spatial relationship features.

[0162] Fig. 9 A schematic diagram of a training method for an image retrieval model provided by an exemplary embodiment of the present application is shown, and the method is applied to Figure 2 The computer device shown is used as an example, and the method includes:

[0163] Step 320, obtaining n sample triplets for training the image retrieval model;

[0164] The computer device obtains n sample triplets for training the image retrieval model.

[0165] For a detailed description of step 320, please refer to the above “Regarding step 320”.

[0166] Step 330, obtaining basic feature vectors of n sample triplets through a basic feature network;

[0167] Through the basic feature network, the computer device obtains the basic feature vectors of the n sample triplets.

[0168] Step 341, obtaining n groups of triplet feature vectors output by the embedding vector network for n sample triples;

[0169] The computer device obtains n groups of triplet feature vectors output by the embedding vector network for n sample triplets.

[0170] Step 342, calculating n first error losses corresponding to n groups of triplet feature vectors;

[0171] The computer device calculates n first error losses corresponding to the n groups of triplet feature vectors.

[0172] Step 343 , from the results of sorting the n first error losses from small to large, select the sample triplets corresponding to the n1 first error losses sorted within the first selected range, and add them to the first sample triplet set for training the quantization index network.

[0173] Among the results of sorting the n first error losses from small to large, the computer device selects sample triplets corresponding to n1 first error losses sorted within the first selected range, and adds them to the first sample triplet set used to train the quantization index network.

[0174] For detailed description of step 341, step 342 and step 343, please refer to Figure 5 The embodiment shown.

[0175] Step 381-1, inputting the first sample triplet set into the quantization index network to obtain n1 third error losses;

[0176] The computer device inputs the first sample triplet set into the quantization index network to obtain n1 third error losses. Wherein, inputting the first sample triplet set into the quantization index network to obtain n1 third error losses includes:

[0177] For each sample triplet of the first sample triplet set, the computer device calculates a second triplet loss of the triplet feature vector; for each sample triplet of the first sample triplet set, the computer device calculates a second quantization error loss of the triplet feature vector; the computer device performs weighted summation of the second triplet loss and the second quantization error loss to obtain a third error loss.

[0178] Step 381 - 2, training a quantization index network based on n1 third error losses;

[0179] The computer device trains a quantized index network based on n1 third error losses.

[0180] Step 361, obtaining n groups of triplet quantization indexes output by the quantization index network for n sample triples;

[0181] The computer device inputs the basic feature vectors of n sample triplets into a quantization index network and outputs n groups of triplet quantization indexes.

[0182] Step 362, calculating n second error losses corresponding to n groups of triplet quantization indices;

[0183] The method for a computer device to calculate n second error losses corresponding to n groups of triplet quantization indices includes:

[0184] For each group of triple quantization indexes, the computer device calculates a first triple loss of the triple quantization indexes; for each group of triple quantization indexes, the computer device calculates a first quantization error loss of the triple quantization indexes; the computer device performs a weighted summation of the first triple loss and the first quantization error loss to obtain a second error loss.

[0185] Step 363, from the n second error losses sorted from small to large, select sample triples corresponding to n2 second error losses sorted within the second selected range, and add them to the second sample triple set for training the embedding vector network;

[0186] The computer device selects sample triplets corresponding to n2 second error losses ranked in the top y% from the results of sorting the n second error losses from small to large, and adds them to the second sample triplet set used to train the quantization index network.

[0187] For detailed description of step 361, step 362 and step 363, please refer to Figure 6 The embodiment shown.

[0188] Step 382-1, inputting the second sample triplet set into the embedding vector network to obtain n2 fourth error losses;

[0189] The computer device inputs the second sample triplet set into the embedding vector network to obtain n2 fourth error losses.

[0190] Step 382-2, training the embedding vector network based on n2 fourth error losses.

[0191] The computer device trains the embedding vector network based on the n2 fourth error loss.

[0192] For detailed description of steps 381-1, 381-2, 382-1 and 382-2, please refer to Figure 7 The embodiment shown.

[0193] In summary, by training the embedding vector network and the quantization index network through sample triplets, not only the first set of sample triplets screened by the embedding vector network is used as the sample triplets for training the quantization index network, but also the second set of sample triplets screened by the quantization index network is used as the sample triplets for training the embedding vector network. The above method enables the embedding vector network to support the calculation of the distance between the feature vectors of the m candidate images and the feature vector of the query image, and also makes the quantization index of the query image obtained by the quantization index network more accurate. The training method of the above image retrieval model eliminates noise sample triplets, and at the same time makes the prediction effects of the quantization index network and the embedding vector network on positive and negative samples similar. The noise sample triplets are predicted by dual branches, and the dual branches learn from each other's excellent sample triplets to achieve denoising learning, so that the dual branches have similar prediction effects.

[0194] Based on the above, the technical solution of the present application has been fully discussed. Next, the relevant training parameters of the image retrieval model will be introduced.

[0195] For the basic feature network, optionally, the basic feature network is a resnet101 network (a convolutional neural network), that is, the basic feature network can be trained using the resnet101 network. The specific parameters are shown in Table 1 below, where Conv1-Conv5 of the basic feature network adopts the parameters of ResNet101 pre-trained on the ImageNet (a large general object recognition open source dataset) dataset. Optionally, the basic feature network can also use a resnet18CNN network.

[0196] Table 1

[0197]

[0198]

[0199] For the embedding vector network, the parameters shown in Table 2 are used for training. The embedding vector network can also be called the Embedding network. The embedding vector network is initialized with a Gaussian distribution with a variance of 0.01 and a mean of 0. Optionally, the embedding vector network can also use multi-layer Fc connections. The embedding vector network outputs a 64-dimensional vector.

[0200] Table 2

[0201]

[0202] For the quantized index network, the parameters shown in Table 3 are used for training. The quantized index network is initialized with a Gaussian distribution with a variance of 0.01 and a mean of 0.

[0203] Table 3

[0204]

[0205] For image retrieval models:

[0206] Setting learning parameters: When updating the image retrieval model, the underlying basic features need to be updated when training the embedding vector network in the first stage. The learning parameters are set as shown in Tables 1 and 2. When training the quantized index network in the second stage, there is no need to update the basic feature network and the embedding vector network.

[0207] Setting the learning rate: The basic feature network, embedding vector network, and quantized index network all use a learning rate of lr1=0.005. After every 10 iterations, lr becomes 0.1 times the original value.

[0208] Set up the learning process: perform epoch iterations for all samples, and process all samples in batches in each iteration until the average epoch loss (the total loss of the image retrieval model obtained by the weighted sum of the third error loss and the fourth error loss) at a certain epoch no longer decreases.

[0209] Set the specific operations in each iteration: divide all similar sample pairs into Nb batches according to batch size (batch-size), and for each batch, obtain the above n sample triplets.

[0210] Set model parameter update: Use SGD (a gradient descent method) stochastic gradient descent method to perform gradient backward calculation on the loss function value obtained in the previous batch to obtain the updated values ​​of the parameters of the embedding vector network and the quantized index network, and update the network.

[0211] In order to achieve image retrieval by building a deep quantization network, Fig.10 is a flowchart of an image retrieval method provided by an exemplary embodiment of the present application. Figure 2 The method is performed by the computer device shown in the figure for example, and the method includes:

[0212] Step 1001, obtaining a basic feature vector of a query image;

[0213] A query image is an image used for image retrieval.

[0214] In one embodiment, the image retrieval model further includes a basic feature network, through which basic feature vectors of n sample triplets are obtained. Schematically, Fig.11 A schematic diagram of a system for using an image retrieval model provided by an exemplary embodiment of the present application is shown, wherein a query image 1101 is input into a basic feature network 403 to obtain a basic feature vector of the query image 1101 .

[0215] Step 1002, inputting the basic feature vector into the quantization index network and the embedding vector network;

[0216] Combined with reference Fig.11 , the basic feature vector of the query image 1101 is input into the quantization index network 402 and the embedding vector network 401.

[0217] Step 1003, obtaining a quantized index of the query image through a quantized index network, and obtaining a feature vector of the query image through an embedded vector network;

[0218] Combined with reference Fig.11 , the quantization index (1, 0, 0) of the query image 1101 is obtained through the quantization index network 402, and the feature vector (0.2, 0.8, 0.3, 0.3) of the query image 1101 is obtained through the embedding vector network 401.

[0219] Step 1004, based on the quantization index, obtain feature vectors of m candidate images from the quantization codebook;

[0220] The quantization codebook stores the mapping relationship between the quantization index and the feature vectors of m candidate images, where m is a positive integer.

[0221] Combined with reference Fig.11 , based on the quantization index (1, 0, 0), the feature vectors of three candidate images are indexed from the quantization codebook: (0.2, 0.7, 0.3, 0.3), (0.1, 0.5, 0.2, 0.2) and (0.2, 0.4, 0.2, 0.3).

[0222] Fig.11 It also shows a construction process of a quantization codebook provided by an exemplary embodiment of the present application, in which features are extracted from all images in the image library to obtain quantization indexes and feature vectors.

[0223] In one embodiment, the quantization codebook is constructed by the following steps:

[0224] First, any image i in the image library is input into the above-mentioned image retrieval model, wherein the feature vector e can be obtained by the embedding vector network 401, and the quantization index q (q is a vector with each dimension of 0 or 1 obtained by a sign function) can be obtained by the quantization index network 402, and a mapping table T[i:e] of the embedding vector network 401 of the image i is recorded (where i represents the sequence number of the image, and e represents the feature vector output by the embedding vector network of the image i).

[0225] Second, record the image numbers with the same q into the mapping table Linvert[i:q] of q, such as {q1: [image 1, image 2, image 5], q2: [image 3], q3: [image 4]}, and save the list of all quantization indexes Lindex: [q1, q2, q3].

[0226] Third, for the image i' newly added to the image library, its qi' and ei' can be calculated. When qi' exists in the list Lindex, i' is directly added to the mapping table Linvert corresponding to qi' under Lindex, and the image serial numbers i' and ei' are added to the T mapping table (a new record of serial number and features is added, such as i':ei').

[0227] The above quantization index list Lindex, q mapping table Linvert and T list together constitute a quantization codebook.

[0228] In one embodiment, based on the quantization index, obtaining feature vectors of m candidate images from the quantization codebook by indexing includes: calculating the quantization index of the query image, determining from the list Lindex several quantization indexes whose Hamming distances with the quantization index of the query image are less than a threshold, and determining from the mapping table Linvert m candidate images corresponding to the several quantization indexes, and determining the feature vectors of the m candidate images through the T mapping table.

[0229] Step 1005, respectively calculating the fifth distances between the feature vectors of the m candidate images and the feature vector of the query image to obtain m fifth distances;

[0230] Combined with reference Fig.11, respectively calculate the fifth distances between the feature vector (0.2, 0.8, 0.3, 0.3) of the query image 1101 and the feature vectors of the three candidate images: (0.2, 0.7, 0.3, 0.3), (0.1, 0.5, 0.2, 0.2) and (0.2, 0.4, 0.2, 0.3), and obtain three fifth distances. The fifth distance is the Euclidean distance.

[0231] Step 1006 , among the m fifth distances sorted from small to large, select the candidate images corresponding to the fifth distances ranked in the top z% according to the preset z value.

[0232] Combined with reference Fig.11 , in the three fifth distances sorted from small to large, according to the preset z value, the candidate images corresponding to the fifth distance ranked in the top z% are screened out, wherein z% can be obtained by pre-configuring the image retrieval model, and optionally, the z value can be reasonably set according to the retrieval requirements so that the screened candidate images meet the retrieval expectations of the image retrieval model. z is a positive number.

[0233] In one embodiment, the computer device further sends the screened candidate images to a client running an image retrieval function.

[0234] In summary, the above method can sort the distances between the query image and m candidate images and filter out the top-ranked images through an image retrieval model that includes a basic feature network, an embedded vector network, and a quantized index network. The above method not only makes the filtered image closer to the query image, but also avoids losing or adding additional candidate images when determining the m candidate images.

[0235] Fig.12 It is a structural block diagram of a training device for an image retrieval model provided by an exemplary embodiment of the present application. The image retrieval model includes an embedding vector network and a quantization index network. The embedding vector network is used to obtain a feature vector of an image. The quantization index network is used to extract a quantization index of an image. The training device for the image retrieval model includes:

[0236] The acquisition module 1201 is used to acquire n sample triplets for training the image retrieval model; the sample triplets include a target sample, a positive sample that forms a similar sample pair with the target sample, and a negative sample that does not form a similar sample pair with the target sample, and n is a positive integer greater than 1;

[0237] A screening module 1202 is used to input the basic feature vectors of the n sample triplets into the embedding vector network; and screen out a first set of sample triplets for training the quantization index network according to the error of the feature vector output by the embedding vector network;

[0238] The screening module 1202 is further used to input the basic feature vectors of the n sample triplets into the quantization index network; and screen out a second sample triplet set for training the embedding vector network according to the error of the quantization index output by the quantization index network;

[0239] The training module 1203 is used to train a quantization index network based on the first sample triplet set, and to train an embedding vector network based on the second sample triplet set.

[0240] In an optional embodiment, the screening module 1202 is further used to obtain n groups of triple feature vectors output by the embedding vector network for n sample triplets.

[0241] In an optional embodiment, the screening module 1202 is further configured to calculate n first error losses corresponding to n groups of triplet feature vectors.

[0242] In an optional embodiment, the screening module 1202 is further used to screen out sample triplets corresponding to n1 first error losses sorted within a first selected range from the results of sorting the n first error losses from small to large, and add them to the first sample triplet set used to train the quantization index network.

[0243] In an optional embodiment, the screening module 1202 is further configured to calculate, for each set of triplet feature vectors, a first distance between the feature vector of the target sample and the feature vector of the positive sample.

[0244] In an optional embodiment, the screening module 1202 is further configured to calculate, for each set of triplet feature vectors, a second distance between the feature vector of the target sample and the feature vector of the negative sample.

[0245] In an optional embodiment, the screening module 1202 is further used to calculate a first error loss between a difference between the first distance and the second distance and a first distance threshold, where the first distance threshold is a threshold of a difference between a distance between a target sample and a positive sample and a distance between a target sample and a negative sample.

[0246] In an optional embodiment, the screening module 1202 is further used to screen out the sample triplets corresponding to the n1 first error losses ranked in the top x% from the sorting results of the n first error losses from small to large according to a preset x value, and add them to the first sample triplet set for training the quantization index network, where n1 is a positive integer less than n. x is a positive number.

[0247] In an optional embodiment, the screening module 1202 is further configured to obtain n groups of triplet quantization indexes output by the quantization index network for the n sample triplets.

[0248] In an optional embodiment, the screening module 1202 is further configured to calculate n second error losses corresponding to the n groups of triplet quantization indices.

[0249] In an optional embodiment, the screening module 1202 is also used to screen out sample triplets corresponding to n2 second error losses sorted within a second selected range from the results of sorting the n second error losses from small to large, and add them to the second sample triplet set used to train the embedding vector network, where n2 is a positive integer less than n.

[0250] In an optional embodiment, the screening module 1202 is further configured to calculate, for each group of triple quantization indexes, a first triple loss of the triple quantization indexes.

[0251] In an optional embodiment, the screening module 1202 is further configured to calculate a first quantization error loss of the triplet quantization index for each group of triplet quantization indexes.

[0252] In an optional embodiment, the screening module 1202 is further configured to perform a weighted summation on the first triplet loss and the first quantization error loss to obtain a second error loss.

[0253] In an optional embodiment, the screening module 1202 is further used to screen out the sample triplets corresponding to the n2 second error losses ranked in the top y% according to a preset y value from the results of sorting the n second error losses from small to large, and add them to the second sample triplet set used to train the quantization index network. y is a positive number.

[0254] In an optional embodiment, the training module 1203 is further configured to input the first sample triplet set into the quantization index network to obtain n1 third error losses.

[0255] In an optional embodiment, the training module 1203 is further used to train the quantization index network based on n1 third error losses.

[0256] In an optional embodiment, the training module 1203 is further configured to calculate a second triplet loss of the triplet feature vector for each sample triplet in the first sample triplet set.

[0257] In an optional embodiment, the training module 1203 is further configured to calculate a second quantization error loss of a triplet feature vector for each sample triplet in the first sample triplet set.

[0258] In an optional embodiment, the training module 1203 is further configured to perform a weighted summation on the second triplet loss and the second quantization error loss to obtain a third error loss.

[0259] In an optional embodiment, the training module 1203 is further used to input the second sample triplet set into the embedding vector network to obtain n2 fourth error losses.

[0260] In an optional embodiment, the training module 1203 is further used to train the embedding vector network based on n2 fourth error losses.

[0261] In an optional embodiment, the training module 1203 is further configured to calculate, for each sample triplet in the second sample triplet set, a third distance between the feature vector of the target sample and the feature vector of the positive sample.

[0262] In an optional embodiment, the training module 1203 is further configured to calculate, for each sample triplet in the second sample triplet set, a fourth distance between a feature vector of the target sample and a feature vector of the negative sample.

[0263] In an optional embodiment, the training module 1203 is further used to calculate a fourth error loss between a difference between the third distance and the fourth distance and a second distance threshold, where the second distance threshold is a threshold of a difference between a distance between a target sample and a positive sample and a distance between a target sample and a negative sample.

[0264] In an optional embodiment, the image retrieval model also includes a basic feature network.

[0265] In an optional embodiment, the acquisition module 1201 is further used to acquire basic feature vectors of n sample triplets through a basic feature network.

[0266] In summary, the training device of the above-mentioned image retrieval model trains the embedding vector network and the quantization index network through sample triplets. Not only the first set of sample triples screened by the embedding vector network is used as the sample triplets for training the quantization index network, but also the second set of sample triples screened by the quantization index network is used as the sample triplets for training the embedding vector network. The training device of the above-mentioned image retrieval model enables the embedding vector network to support the calculation of the distance between the feature vectors of m candidate images and the feature vector of the query image, and also makes the quantization index of the query image obtained by the quantization index network more accurate.

[0267] Fig.13 The structure block diagram of an image retrieval device provided by an exemplary embodiment of the present application is shown, and the device includes:

[0268] An acquisition module 1301 is used to acquire a basic feature vector of a query image;

[0269] An input module 1302, used to input the basic feature vector into the quantization index network and the embedding vector network;

[0270] The acquisition module 1301 is further used to acquire the quantization index of the query image through the quantization index network, and acquire the feature vector of the query image through the embedding vector network;

[0271] An indexing module 1303 is used to obtain feature vectors of m candidate images from a quantization codebook based on a quantization index; the quantization codebook stores a mapping relationship between the quantization index and the feature vectors of the m candidate images, where m is a positive integer;

[0272] A calculation module 1304 is used to calculate the fifth distances between the feature vectors of the m candidate images and the feature vector of the query image respectively, to obtain m fifth distances;

[0273] The screening module 1305 is used to screen out the candidate images corresponding to the fifth distances ranked in the top z% from the m fifth distances sorted from small to large according to a preset z value, where z is a positive number.

[0274] In an optional embodiment, the acquisition module 1301 is further used to generate a basic feature vector of the query image through a basic feature network of the image retrieval model.

[0275] To summarize, the above-mentioned image retrieval device can sort the distances between the query image and m candidate images and filter out the top-ranked images through an image retrieval model that includes a basic feature network, an embedded vector network, and a quantized index network. The above-mentioned image retrieval device not only makes the filtered images closer to the query image, but also avoids losing or adding additional candidate images when determining the m candidate images.

[0276] It should be noted that the image retrieval training device and image retrieval device provided in the above embodiments are only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image retrieval model training device provided in the above embodiments belongs to the same concept as the image retrieval model training method embodiment, and the image retrieval device and the image retrieval method belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0277] Fig.14 The block diagram of the structure of a computer device 1400 provided by an exemplary embodiment of the present application is shown. The computer device may be a terminal or a server, and in this embodiment, it can be simply described as the terminal training an image retrieval model alone and / or the terminal using an image retrieval model alone, or the server training an image retrieval model alone and / or the server using an image retrieval model alone, or the terminal and the server jointly training an image retrieval model and / or the terminal and the server jointly using an image retrieval model.

[0278] Typically, the computer device 1400 includes a processor 1401 and a memory 1402 .

[0279] The processor 1401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1401 may also include an AI processor, which is used to process computing operations related to machine learning.

[0280] The memory 1402 may include one or more computer-readable storage media, which may be non-transitory. The memory 1402 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1402 is used to store at least one instruction, which is used to be executed by the processor 1401 to implement the training method or image retrieval method of the image retrieval model provided in the method embodiment of the present application.

[0281] In some embodiments, the computer device 1400 may further optionally include: a peripheral device interface 1403 and at least one peripheral device. The processor 1401, the memory 1402 and the peripheral device interface 1403 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1403 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1404, a display screen 1405, a camera assembly 1406, an audio circuit 1407 and a power supply 1408.

[0282] The peripheral device interface 1403 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1401 and the memory 1402. In some embodiments, the processor 1401, the memory 1402, and the peripheral device interface 1403 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1401, the memory 1402, and the peripheral device interface 1403 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0283] The radio frequency circuit 1404 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1404 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1404 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1404 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1404 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1404 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0284] The display screen 1405 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1405 is a touch display screen, the display screen 1405 also has the ability to collect touch signals on the surface or above the surface of the display screen 1405. The touch signal can be input to the processor 1401 as a control signal for processing. At this time, the display screen 1405 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1405 can be one, set on the front panel of the computer device 1400; in other embodiments, the display screen 1405 can be at least two, respectively set on different surfaces of the computer device 1400 or in a folding design; in other embodiments, the display screen 1405 can be a flexible display screen, set on a curved surface or a folding surface of the computer device 1400. Even, the display screen 1405 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1405 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0285] The camera assembly 1406 is used to capture images or videos. Optionally, the camera assembly 1406 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1406 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0286] The audio circuit 1407 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1401 for processing, or input them into the radio frequency circuit 1404 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the computer device 1400. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1401 or the radio frequency circuit 1404 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1407 may also include a headphone jack.

[0287] The power supply 1408 is used to power various components in the computer device 1400. The power supply 1408 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1408 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0288] In some embodiments, the computer device 1400 further includes one or more sensors 1409 , including but not limited to: an acceleration sensor 1410 , a gyroscope sensor 1411 , a pressure sensor 1412 , an optical sensor 1414 , and a proximity sensor 1415 .

[0289] The acceleration sensor 1410 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the computer device 1400. For example, the acceleration sensor 1410 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1401 can control the display screen 1405 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1410. The acceleration sensor 1410 can also be used to collect game or user motion data.

[0290] The gyro sensor 1411 can detect the body direction and rotation angle of the computer device 1400, and the gyro sensor 1411 can cooperate with the acceleration sensor 1410 to collect the user's 3D actions on the computer device 1400. The processor 1401 can implement the following functions based on the data collected by the gyro sensor 1411: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0291] The pressure sensor 1412 can be set on the side frame of the computer device 1400 and / or the lower layer of the display screen 1405. When the pressure sensor 1412 is set on the side frame of the computer device 1400, it can detect the user's grip signal of the computer device 1400, and the processor 1401 performs left and right hand recognition or shortcut operations according to the grip signal collected by the pressure sensor 1412. When the pressure sensor 1412 is set on the lower layer of the display screen 1405, the processor 1401 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 1405. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0292] The optical sensor 1413 is used to collect the ambient light intensity. In one embodiment, the processor 1401 can control the display brightness of the display screen 1405 according to the ambient light intensity collected by the optical sensor 1413. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1405 is increased; when the ambient light intensity is low, the display brightness of the display screen 1405 is reduced. In another embodiment, the processor 1401 can also dynamically adjust the shooting parameters of the camera assembly 1406 according to the ambient light intensity collected by the optical sensor 1413.

[0293] The proximity sensor 1414, also called a distance sensor, is usually disposed on the front panel of the computer device 1400. The proximity sensor 1414 is used to collect the distance between the user and the front of the computer device 1400. In one embodiment, when the proximity sensor 1414 detects that the distance between the user and the front of the computer device 1400 is gradually decreasing, the processor 1401 controls the display screen 1405 to switch from the screen-on state to the screen-off state; when the proximity sensor 1414 detects that the distance between the user and the front of the computer device 1400 is gradually increasing, the processor 1401 controls the display screen 1405 to switch from the screen-off state to the screen-on state.

[0294] Those skilled in the art will understand that Fig.14 The structure shown in the figure does not constitute a limitation on the computer device 1400, and the computer device 1400 may include more or less components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0295] The present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the image retrieval model training method or image retrieval method provided in the above method embodiment.

[0296] The present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the training method of the image retrieval model or the image retrieval method provided by the above method embodiment.

[0297] In one embodiment, the sample triples, the basic feature vectors output by the basic feature network, the feature vectors output by the embedding vector network, and the quantized indexes output by the quantized index network can be stored in the nodes of the data sharing system. Fig.15 The data sharing system shown, the data sharing system 1500 refers to a system for sharing data between nodes, and the data sharing system may include multiple nodes 1501, and the multiple nodes 1501 may refer to each client in the data sharing system. Each node 1501 can receive input information when performing normal work, and maintain the shared data in the data sharing system based on the received input information. In order to ensure the information intercommunication within the data sharing system, there may be an information connection between each node in the data sharing system, and information can be transmitted between nodes through the above information connection. For example, when any node in the data sharing system receives input information, other nodes in the data sharing system obtain the input information according to the consensus algorithm, and store the input information as data in the shared data, so that the data stored on all nodes in the data sharing system are consistent.

[0298] Each node in the data sharing system has a node identifier corresponding to it, and each node in the data sharing system can store the node identifiers of other nodes in the data sharing system, so that the generated blocks can be broadcast to other nodes in the data sharing system according to the node identifiers of other nodes. A node identifier list as shown in the following table can be maintained in each node, and the node name and node identifier are stored in the node identifier list accordingly. Among them, the node identifier can be an IP (Internet Protocol, a protocol for interconnecting networks) address and any other information that can be used to identify the node. Table 4 only takes the IP address as an example for explanation.

[0299] Table 4

[0300]

[0301] Each node in the data sharing system stores the same blockchain. The blockchain consists of multiple blocks, see Fig.16The blockchain consists of multiple blocks. The genesis block includes a block header and a block body. The block header stores the input information characteristic value, version number, timestamp and difficulty value, and the block body stores the input information; the next block of the genesis block takes the genesis block as the parent block, and the next block also includes a block header and a block body. The block header stores the input information characteristic value of the current block, the block header characteristic value, version number, timestamp and difficulty value of the parent block, and so on, so that the block data stored in each block in the blockchain is associated with the block data stored in the parent block, ensuring the security of the input information in the block.

[0302] When generating each block in the blockchain, see Fig.17 When the node where the blockchain is located receives the input information, it verifies the input information. After the verification is completed, the input information is stored in the memory pool and the hash tree used to record the input information is updated. After that, the update timestamp is updated to the time when the input information is received, and different random numbers are tried to calculate the eigenvalue multiple times so that the calculated eigenvalue can satisfy the following formula:

[0303] SHA256(SHA256(version+prev_hash+merkle_root+ntime+nbits+x)) <TARGET

[0304] Among them, SHA256 is the eigenvalue algorithm used to calculate the eigenvalue; version (version number) is the version information of the relevant block protocol in the blockchain; prev_hash is the block header eigenvalue of the parent block of the current block; merkle_root is the eigenvalue of the input information; ntime is the update time of the update timestamp; nbits is the current difficulty, which is a fixed value within a period of time and is determined again after exceeding the fixed time period; x is a random number; TARGET is the eigenvalue threshold, which can be determined based on nbits.

[0305] In this way, when the random number that satisfies the above formula is calculated, the information can be stored accordingly, the block header and block body can be generated, and the current block can be obtained. Subsequently, the node where the blockchain is located sends the newly generated block to other nodes in the data sharing system according to the node identification of other nodes in the data sharing system. Other nodes verify the newly generated block and add the newly generated block to the blockchain stored in them after the verification is completed.

[0306] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0307] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0308] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A training method for an image retrieval model, characterized in that: The image retrieval model includes an embedding vector network and a quantization index network, wherein the embedding vector network is used to obtain a feature vector of an image, and the quantization index network is used to extract a quantization index of the image; the method includes: Obtain n sample triplets for training the image retrieval model; the sample triplets include a target sample, a positive sample that forms a similar sample pair with the target sample, and a negative sample that does not form a similar sample pair with the target sample, where n is a positive integer greater than 1; Input the basic feature vectors of the n sample triplets into the embedding vector network; obtain n groups of triple feature vectors output by the embedding vector network for the n sample triplets; for each group of the triple feature vectors, calculate the first distance between the feature vector of the target sample and the feature vector of the positive sample; for each group of the triple feature vectors, calculate the second distance between the feature vector of the target sample and the feature vector of the negative sample; calculate the first error loss between the difference between the first distance and the second distance and a first distance threshold, the first distance threshold being the threshold of the difference between the distance between the target sample and the positive sample and the distance between the target sample and the negative sample; from the sorting results of the n first error losses corresponding to the n groups of triple feature vectors from small to large, filter out the sample triplets corresponding to the n1 first error losses sorted within the first selected range, and add them to the first sample triple set for training the quantization index network, where n1 is a positive integer less than n; Input the basic feature vectors of the n sample triplets into the quantization index network; obtain n groups of triple quantization indexes output by the quantization index network for the n sample triplets; for each group of triple quantization indexes, calculate the first triple loss of the triple quantization index; for each group of triple quantization indexes, calculate the first quantization error loss of the triple quantization index; perform weighted summation of the first triple loss and the first quantization error loss to obtain a second error loss; from the sorting results of the n second error losses corresponding to the n groups of triple quantization indexes from small to large, screen out the sample triplets corresponding to the n2 second error losses sorted within the second selection range, and add them to the second sample triple set used to train the embedding vector network, where n2 is a positive integer less than n; The quantization index network is trained based on the first set of sample triples, and the embedding vector network is trained based on the second set of sample triples.

2. The method according to claim 1, characterized in that The method of selecting sample triples corresponding to n1 first error losses sorted within a first selected range from the results of sorting the n first error losses from small to large, and adding them to the first sample triple set for training the quantization index network, comprises: Among the results of sorting the n first error losses from small to large, according to a preset x value, sample triplets corresponding to the n1 first error losses ranked in the top x% are screened out and added to the first sample triplet set used to train the quantization index network, where x is a positive number.

3. The method according to claim 1, characterized in that The method of selecting sample triples corresponding to n2 second error losses sorted within a second selected range from the results of sorting the n second error losses from small to large, and adding them to the second sample triple set for training the embedding vector network, comprises: Among the results of sorting the n second error losses from small to large, according to the preset y value, the sample triplets corresponding to the n2 second error losses ranked in the top y% are screened out and added to the second sample triplet set used to train the quantization index network, where y is a positive number.

4. The method according to any one of claims 1 to 3, characterized in that: The step of training the quantization index network based on the first sample triplet set includes: For each of the sample triples of the first set of sample triples, calculating a second triplet loss of a triplet feature vector by the quantization index network, the triplet feature vector being a feature vector output by the embedding vector network, the first set of sample triples including n1 sample triplets; For each of the sample triples in the first set of sample triples, calculating a second quantization error loss of the triplet feature vector by the quantization index network; Performing a weighted summation on the second triplet loss and the second quantization error loss to obtain a third error loss; The quantization index network is trained based on n1 third error losses.

5. The method according to any one of claims 1 to 3, characterized in that: The step of training the embedding vector network based on the second sample triplet set includes: For each of the sample triples in the second sample triplet set, calculating a third distance between a feature vector of the target sample and a feature vector of the positive sample by using the embedding vector network, the second sample triplet set including n2 sample triples; For each of the sample triples of the second sample triplet set, calculating, by the embedding vector network, a fourth distance between a feature vector of the target sample and a feature vector of the negative sample; Calculating a fourth error loss between a difference between the third distance and the fourth distance and a second distance threshold, wherein the second distance threshold is a threshold of a difference between a distance between the target sample and the positive sample and a distance between the target sample and the negative sample; The embedding vector network is trained based on n2 fourth error losses.

6. The method according to claim 1, characterized in that The image retrieval model also includes a basic feature network; The method further comprises: The basic feature vectors of the n sample triplets are obtained through the basic feature network.

7. An image retrieval method, characterized in that: The method is applied to the image retrieval model trained by any one of claims 1 to 6, and the method comprises: Get the basic feature vector of the query image; Inputting the basic feature vector into the quantization index network and the embedding vector network; Acquire a quantized index of the query image through the quantized index network, and acquire a feature vector of the query image through the embedded vector network; Based on the quantization index, obtain feature vectors of m candidate images from a quantization codebook; the quantization codebook stores a mapping relationship between the quantization index and the feature vectors of the m candidate images, where m is a positive integer; respectively calculating fifth distances between the feature vectors of the m candidate images and the feature vector of the query image to obtain m fifth distances; Among the m fifth distances sorted from small to large, the candidate images corresponding to the fifth distances ranked in the top z% are screened out according to a preset z value, where z is a positive number.

8. A training device for an image retrieval model, characterized in that: The image retrieval model includes an embedding vector network and a quantization index network, wherein the embedding vector network is used to obtain a feature vector of an image, and the quantization index network is used to extract a quantization index of the image; the device includes: An acquisition module, used for acquiring n sample triplets for training the image retrieval model; the sample triplets include a target sample, a positive sample that forms a similar sample pair with the target sample, and a negative sample that does not form a similar sample pair with the target sample, and n is a positive integer greater than 1; A screening module, used to input the basic feature vectors of the n sample triplets into the embedding vector network; obtain n groups of triple feature vectors output by the embedding vector network for the n sample triplets; for each group of the triple feature vectors, calculate the first distance between the feature vector of the target sample and the feature vector of the positive sample; for each group of the triple feature vectors, calculate the second distance between the feature vector of the target sample and the feature vector of the negative sample; calculate the first error loss between the difference between the first distance and the second distance and a first distance threshold, the first distance threshold being the threshold of the difference between the distance between the target sample and the positive sample and the distance between the target sample and the negative sample; from the sorting results of the n first error losses corresponding to the n groups of triple feature vectors from small to large, screen out the sample triplets corresponding to the n1 first error losses sorted within the first selected range, and add them to the first sample triple set used to train the quantization index network, where n1 is a positive integer less than n; The screening module is further used to input the basic feature vectors of the n sample triplets into the quantization index network; obtain n groups of triple quantization indexes output by the quantization index network for the n sample triplets; for each group of triple quantization indexes, calculate the first triple loss of the triple quantization index; for each group of triple quantization indexes, calculate the first quantization error loss of the triple quantization index; perform weighted summation of the first triple loss and the first quantization error loss to obtain a second error loss; from the sorting results of the n second error losses corresponding to the n groups of triple quantization indexes from small to large, screen out the sample triplets corresponding to the n2 second error losses sorted within the second selected range, and add them to the second sample triple set used to train the embedding vector network, where n2 is a positive integer less than n; A training module is used to train the quantization index network based on the first set of sample triples, and to train the embedding vector network based on the second set of sample triples.

9. An image retrieval device, characterized in that: The device is applied to the image retrieval model trained by any one of claims 1 to 6, and the device comprises: An acquisition module, used to acquire a basic feature vector of a query image; An input module, used for inputting the basic feature vector into the quantization index network and the embedding vector network; The acquisition module is further used to acquire the quantization index of the query image through the quantization index network, and acquire the feature vector of the query image through the embedding vector network; An indexing module, configured to obtain feature vectors of m candidate images from a quantization codebook based on the quantization index; the quantization codebook stores a mapping relationship between the quantization index and the feature vectors of the m candidate images, where m is a positive integer; a calculation module, used for respectively calculating fifth distances between the feature vectors of the m candidate images and the feature vector of the query image, to obtain m fifth distances; The screening module is used to screen out the candidate images corresponding to the fifth distances ranked in the top z% from the m fifth distances sorted from small to large according to a preset z value, where z is a positive number.

10. A computer device, characterized in that: The computer device includes: a processor and a memory, the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the training method of the image retrieval model as described in any one of claims 1 to 6, and / or the image retrieval method as described in claim 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the training method of the image retrieval model as described in any one of claims 1 to 6, and / or the image retrieval method as described in claim 7.

Images