Image retrieval model training method and device, equipment and storage medium

Abstract

Embodiments of the present application provide a method and device for training an image retrieval model, and a storage medium. After obtaining hash codes of a plurality of sample images by using the image retrieval model, a plurality of class centers corresponding to a plurality of image categories to which the plurality of sample images belong are obtained, and the image retrieval model and the plurality of class centers are trained based on distances from each sample image to the class center corresponding to the sample image and distances between the plurality of class centers. This training method can reduce the computational complexity of model training, thereby improving the training efficiency of the model. Moreover, the trained image retrieval model can generate hash codes with better discrimination, thereby improving the accuracy of the image retrieval model while improving the training efficiency.

Description

Technical Field

[0001] The present application relates to the field of image retrieval technology, and in particular to a training method, apparatus, device and storage medium for an image retrieval model. Background Art

[0002] With the continuous development of deep learning technology, deep learning-based image retrieval models have been widely used in many scenarios, including entertainment, security, and information security. To quickly and efficiently search large-scale image libraries, image retrieval models often incorporate hash functions, which convert images into binary hash codes. These functions leverage the efficiency of binary hash codes in Hamming distance calculations and their advantages in storage space to achieve large-scale image retrieval.

[0003] However, the computational complexity of related technologies in training image retrieval models is too large. Developers often select some samples from the entire training set for training, which makes the image retrieval model unable to fully learn the information of all samples, resulting in the trained image retrieval model having low accuracy in image retrieval. Summary of the Invention

[0004] The present invention provides a method, apparatus, device, and storage medium for training an image retrieval model, which can improve training efficiency while also increasing the accuracy of the image retrieval model. The technical solution is as follows:

[0005] In one aspect, a method for training an image retrieval model is provided, the method comprising:

[0006] Processing a plurality of sample images based on an image retrieval model to obtain hash codes of the plurality of sample images;

[0007] Based on the multiple image categories to which the multiple sample images belong, obtaining class centers of the multiple image categories;

[0008] Calculating a loss value based on first information between the multiple sample images and the class centers corresponding to the multiple sample images, and second information between the class centers, where the first information and the second information are respectively used to indicate information of hash codes of the corresponding sample images in the Hamming space;

[0009] Based on the loss value, the image retrieval model and each class center are trained.

[0010] In another aspect, a training device for an image retrieval model is provided, the device comprising:

[0011] A processing module, configured to process a plurality of sample images based on an image retrieval model to obtain hash codes of the plurality of sample images;

[0012] a class center acquisition module, configured to acquire class centers of the multiple image categories based on the multiple image categories to which the multiple sample images belong;

[0013] a calculation module, configured to calculate a loss value based on first information between the plurality of sample images and the class centers corresponding to the plurality of sample images, and second information between the class centers, wherein the first information and the second information are respectively used to indicate information of hash codes of the corresponding sample images in a Hamming space;

[0014] A training module is used to train the image retrieval model and each class center based on the loss value.

[0015] In an optional implementation, the device further includes:

[0016] A regression matrix acquisition module, configured to acquire a regression matrix based on a label matrix and hash codes of the plurality of sample images, wherein the label matrix is ​​used to indicate the image categories to which the plurality of sample images belong;

[0017] This calculation module is used to:

[0018] The loss value is calculated based on first information between the multiple sample images and the class centers corresponding to the multiple sample images, second information between the respective class centers, and the regression matrix.

[0019] In an optional implementation, the calculation module is used to:

[0020] Keeping the regression matrix, the network parameters of the image retrieval model, and the centers of each class unchanged, the hash codes of the multiple sample images are updated to obtain updated hash codes of the multiple sample images;

[0021] Keeping the regression matrix and the updated hash codes of the multiple sample images unchanged, calculating the loss value based on the first information between the multiple sample images and the class centers corresponding to the multiple sample images, the second information between the class centers, and the regression matrix;

[0022] The training module is used to update the network parameters of the image retrieval model and each of the class centers based on the loss value to obtain an updated image retrieval model and each of the class centers.

[0023] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed in the training method of the image retrieval model in the embodiment of the present application.

[0024] On the other hand, a computer-readable storage medium is provided, which stores at least one computer program, and the at least one computer program is loaded and executed by a processor to implement the operations performed in the training method of the image retrieval model in the embodiment of the present application.

[0025] In another aspect, a computer program product or computer program is provided, comprising computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the image retrieval model training method provided in the various optional implementations described above.

[0026] An embodiment of the present application provides a method for training an image retrieval model. After obtaining hash codes for multiple sample images through the image retrieval model, the class centers corresponding to the multiple image categories to which the multiple sample images belong are obtained. The image retrieval model and each class center are then trained based on the distance between each sample image and its corresponding class center, as well as the distance between each class center. This method of training an image retrieval model can reduce the computational complexity of model training, thereby improving the training efficiency of the model. Moreover, the trained image retrieval model can generate more discriminative hash codes, thereby effectively improving the accuracy of the image retrieval model while improving training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0028] Figure 1 Schematic diagram of an implementation environment for a training method for an image retrieval model provided in an embodiment of the present application;

[0029] Figure 2 is a structural diagram of an image retrieval model provided according to an embodiment of the present application;

[0030] Figure 3 This is a flowchart of a method for training an image retrieval model according to an embodiment of the present application;

[0031] Figure 4 is a flowchart of another image retrieval model training method provided according to an embodiment of the present application;

[0032] Figure 5is a flowchart of another image retrieval model training method provided according to an embodiment of the present application;

[0033] Figure 6 1 is a schematic diagram of a training architecture of an image retrieval model provided according to an embodiment of the present application;

[0034] Figure 7 Schematic diagram of experimental results of various solutions provided in the embodiments of the present application;

[0035] Figure 8 Schematic diagram of experimental results of another embodiment of the present invention;

[0036] Figure 9 Schematic diagram of experimental results of another embodiment of the present invention;

[0037] Figure 10 1 is a schematic structural diagram of a training device for an image retrieval model provided in accordance with an embodiment of the present application;

[0038] Figure 11 It is a structural diagram of a server provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0040] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0041] In this application, the terms "first," "second," and the like are used to distinguish identical or similar items having substantially the same role and function. It should be understood that "first," "second," and "nth" do not have a logical or temporal dependency, nor do they limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," and the like to describe various elements, these elements should not be limited by these terms.

[0042] These terms are only used to distinguish one element from another. For example, a first image can be referred to as a second image, and similarly, a second image can be referred to as a first image without departing from the scope of various examples. The first image and the second image can both be images, and in some cases, can be separate and different images.

[0043] Here, at least one refers to one or more than one. For example, at least one image can be one image, two images, three images, or any other integer greater than or equal to one. A plurality refers to two or more than two. For example, a plurality of images can be two images, three images, or any other integer greater than or equal to two.

[0044] The following introduces the technologies involved in the training scheme of the image retrieval model provided in the embodiments of this application.

[0045] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0046] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0047] Computer vision (CV) is the study of how machines can "see." Specifically, it involves using cameras and computers to replace the human eye in identifying, tracking, and measuring objects. This is followed by further image processing, which allows the computer to create images more suitable for human observation or for transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data.

[0048] Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, three-dimensional technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies. It also includes common biometric recognition technologies such as face recognition and fingerprint recognition.

[0049] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0050] The following introduces the key terms or abbreviations involved in the training scheme of the image retrieval model provided in the embodiment of the present application.

[0051] A hash algorithm uses a hash function to map data in a high-dimensional space into a compact, fixed-length binary hash code. Hash codes offer the advantages of lightweight storage, efficient XOR operations, and the ability to preserve the original data's information in a high-dimensional space.

[0052] Hamming distance represents the number of different characters in corresponding positions of two strings of the same length, that is, the number of characters that need to be replaced to transform one string into another.

[0053] Hamming space is a metric space composed of binary strings through Hamming distance.

[0054] The following introduces the implementation environment of the image retrieval model training method provided in the embodiment of the present application.

[0055] Figure 1 1 is a schematic diagram of an implementation environment of the image retrieval model training method provided in an embodiment of the present application. The implementation environment includes: a terminal 101 and a server 102.

[0056] The terminal 101 and the server 102 can be connected directly or indirectly via a wired network or a wireless network, which is not limited in this application. Optionally, the terminal 101 is a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The terminal 101 can provide the server 102 with information required for the training method of the image retrieval model, such as training parameters, sample images, and a label matrix.

[0057] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Server 102 is used to execute the image retrieval model training method provided in the embodiment of the present application, and trains the image retrieval model based on the information provided by terminal 101.

[0058] Optionally, during the training of the image retrieval model, the server 102 undertakes the main computing work and the terminal 101 undertakes the secondary computing work; or, the server 102 undertakes the secondary computing work and the terminal 101 undertakes the main computing work; or, the server 102 or the terminal 101 can each undertake the computing work independently.

[0059] Optionally, terminal 101 generally refers to one of multiple terminals. This embodiment uses terminal 101 as an example. Those skilled in the art will appreciate that the number of terminals 101 can be greater. For example, if there are dozens, hundreds, or even more terminals 101, the implementation environment of the image retrieval model training method may also include other terminals. This embodiment of the application does not limit the number of terminals or device types.

[0060] Optionally, the above-mentioned wired network or wireless network uses standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network. In some embodiments, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0061] In order to more clearly illustrate the training method of the image retrieval model provided in the present application, the structure of the image retrieval model provided in the embodiments of the present application is illustrated below.

[0062] Figure 2 Schematic diagram of an image retrieval model provided according to an embodiment of the present application. Figure 2 As shown, the image retrieval model is a model built based on a convolutional neural network (CNN). The image retrieval model 200 includes: an input layer 201, a CNN layer 202, a first fully connected layer 203 and a second fully connected layer 204.

[0063] The input layer 201 is used to input images into the image retrieval model. The CNN layer 202 is used to extract features from the image to obtain high-dimensional image features. For example, the dimension of the high-dimensional image features is 2176. The first fully connected layer 203 is used to perform feature learning on the high-dimensional image features output by the CNN layer 202 to obtain learned image features. For example, the dimension of the learned image features is 1024. The second fully connected layer 204 is used to map the learned image features to a binary range based on a hash function and output a hash code of length l, where l is a positive integer. This entire process can also be referred to as deep hash learning.

[0064] Of course, the structure of the above-mentioned image retrieval model is shown for exemplary purposes only. In some embodiments, the image retrieval model may also have other structures. For example, the image retrieval model is a model based on a deep neural network (DNN) or a recurrent neural network (RNN). The present embodiment does not limit the structure of the image retrieval model.

[0065] The following examples illustrate the training method of the image retrieval model provided by this application.

[0066] Figure 3 is a flowchart of a method for training an image retrieval model according to an embodiment of the present application, such as Figure 3 As shown, in the embodiment of the present application, the application is taken as an example to illustrate the method. The method includes the following steps:

[0067] 301. The server processes multiple sample images based on an image retrieval model to obtain hash codes of the multiple sample images.

[0068] In an embodiment of the present application, a sample image is an image annotated with label information, and the label information is used to identify the features of the sample image. For example, if the sample image is a face image, the label information includes but is not limited to the facial features of the face. For another example, if the sample image is an animal image, the label information includes but is not limited to the species features, hair features, and torso features of the animal. For another example, if the sample image is a vehicle image, the label information includes but is not limited to the species features, brand features, and color features of the vehicle. The embodiment of the present application does not limit the specific type of the sample image.

[0069] 302. The server obtains class centers of multiple image categories based on the multiple image categories to which the multiple sample images belong.

[0070] In an embodiment of the present application, the number of categories of the multiple image categories is a positive integer, one image category includes at least one sample image, and the image corresponding to the class center of each image category is the most representative image in the image category.

[0071] 303. The server calculates a loss value based on first information between multiple sample images and class centers corresponding to the multiple sample images and second information between each class center, where the first information and the second information are respectively used to indicate the distance between the hash codes of the corresponding sample images in the Hamming space.

[0072] In the embodiment of the present application, the first information is used to indicate the distance between the hash code of a sample image and the hash code of its corresponding class center in the Hamming space. The second information is used to indicate the distance between the hash codes of any two class centers in the Hamming space.

[0073] 304. The server trains the image retrieval model and each class center based on the loss value.

[0074] The present application provides a method for training an image retrieval model. After obtaining hash codes for multiple sample images through the image retrieval model, the class centers corresponding to the multiple image categories to which the multiple sample images belong are obtained. The image retrieval model and each class center are then trained based on the distance between each sample image and its corresponding class center, as well as the distance between each class center. This training method can reduce the computational complexity of model training, thereby improving the training efficiency of the model. Moreover, the trained image retrieval model can generate more discriminative hash codes, thereby effectively improving the accuracy of the image retrieval model while improving training efficiency.

[0075] According to the above Figure 3 The embodiment shown briefly describes the training method of the image retrieval model provided by the present application. The following describes the training method of the image retrieval model provided by the present application in detail based on several specific implementation methods.

[0076] It should be noted that, in the following embodiments, the following expressions are used as examples for description:

[0077] The image retrieval model is denoted as f(·).

[0078] The training dataset X is represented as Where N is the number of samples, N is a positive integer, and i is a positive integer.

[0079] The predictive code H is expressed as

[0080] The hash code B is expressed as Where l is the hash code length.

[0081] The label matrix Y is expressed as Y is defined as follows: If x i belongs to category j, then y ji =1, otherwise y ji =0, where C is the number of categories in the training data set and j is a positive integer.

[0082] The class center M is expressed as in, It is represented as the class center of the j-th category, expressed as a hash code.

[0083] Of course, the above expressions are only shown for exemplary description. In some embodiments, the above expressions can also be in other forms, which is not limited in the embodiments of the present application.

[0084] It should be noted that during the training process of the image retrieval model, two types of parameters are involved: one type is the parameters to be trained, referred to as network parameters in the following embodiments, and the other type is hyperparameters, which refer to parameters set before the learning process begins in machine learning. Optionally, the hyperparameters of the image retrieval model include, but are not limited to, training cycle, batch size, hash code length, and initial learning rate. This embodiment of the present application does not limit this.

[0085] In addition, during the training process of the image retrieval model, the server performs batch training on the training data set and updates the network parameters of the image retrieval model through multiple iterative training until the target training conditions are met. Optionally, the target training condition refers to the loss value of the image retrieval model being less than or equal to a loss value threshold, or the target training condition refers to the number of iterations reaching a threshold number. This embodiment of the present application is not limited to this.

[0086] Schematically, the server determines the number of sample images participating in each iterative training according to the batch sampling size m (m is a positive integer), and determines the number of iterations (also known as training batches) based on the number of sample images in the training dataset. Taking any iterative training as an example, the server inputs m sample images into the image retrieval model, obtains the corresponding loss value of the model, and updates the network parameters of the image retrieval model based on the loss value. The server then performs the next iterative training based on the updated image retrieval model until the target training conditions are met, resulting in a trained image retrieval model.

[0087] The following combination Figure 4 A specific implementation of the image retrieval model training method provided in this application is described. Figure 4 is a flowchart of another image retrieval model training method provided in an embodiment of the present application, such as Figure 4As shown, the training method of the image retrieval model is applied to the server. The following is an example of the server performing the first iteration of training. The method includes the following steps:

[0088] 401. The server calls an image retrieval model to process multiple sample images to obtain predictive codes of the multiple sample images.

[0089] In an embodiment of the present application, the server initializes the image retrieval model, configures the network parameters of the image retrieval model as initialization parameters, and then inputs multiple sample images into the image retrieval model based on the hyperparameters of the image retrieval model. The image retrieval model processes the multiple sample images and outputs predicted codes of the multiple sample images.

[0090] Optionally, the server performs Glorot uniform distribution initialization and configures the network parameters of the image retrieval model as initialization parameters. Optionally, the terminal provides a setting function for initialization parameters, and the developer can set the initialization parameters through the terminal. In response to the developer's submission operation of the initialization parameters, the terminal sends the initialization parameters to the server, and the server initializes the image retrieval model based on the received initialization parameters. The embodiments of the present application do not limit the initialization method of the image retrieval model.

[0091] Optionally, the server is associated with an image database, which is used to store a training dataset for the image retrieval model. Optionally, the image database stores the training dataset based on blockchain. Schematically, the server can extract the training dataset from the image database, select multiple sample images from the training dataset, and input the multiple sample images into the image retrieval model. For example, the server extracts the training dataset from the image database. If the batch sampling size is m, the server randomly selects m sample images from the training data set X, inputs these m sample images into the image retrieval model, and obtains the prediction codes corresponding to the m sample images.

[0092] 402. The server performs binarization processing on the predicted codes of the multiple sample images to obtain hash codes of the multiple sample images.

[0093] In the embodiment of the present application, the server performs binarization processing on the prediction codes of the multiple sample images and converts the prediction codes of the multiple sample images into hash codes respectively. The binarization processing refers to calling a binarization function to map the prediction codes of the multiple sample images into a binary space. Schematically, the binarization function is a sign function sgn(·). For example, the prediction codes of the multiple sample images are Hash codes for multiple sample images Among them, b i =sgn(h i).

[0094] 403. The server obtains class centers of multiple image categories based on the multiple image categories to which the multiple sample images belong.

[0095] In an embodiment of the present application, the server classifies sample images in a training dataset based on a label matrix to obtain initial class centers corresponding to the training dataset. The server then determines the image categories to which multiple sample images belong based on the label matrix, and obtains class centers for multiple image categories from the initial class centers based on the multiple image categories to which the multiple sample images belong. The label matrix can indicate the image category to which any sample image in the training dataset belongs.

[0096] Optionally, after classifying the sample images in the training dataset based on the label matrix, the server performs a standard normal distribution on the sample images in each image category, and determines the initial class center corresponding to each image category based on the standard normal distribution results. The embodiment of the present application does not limit the method for determining the initial class center.

[0097] In addition, in the embodiment of the present application, the server determines the initial cluster center when executing step 403. In some embodiments, the server determines the initial cluster center while executing steps 401 and 402. In other embodiments, the server determines the initial cluster center before executing step 401. This embodiment of the present application is not limited to this.

[0098] It should be noted that in the embodiments of the present application, the server executes steps 401 to 403 in the order in which they are executed. In some embodiments, after the server executes step 401, it first executes step 403 and then executes step 402. In other embodiments, after the server executes step 401, it executes steps 402 and 403 simultaneously. The embodiments of the present application do not limit the order in which steps 401 to 403 are executed.

[0099] 404. The server obtains first information between the multiple sample images and the class centers corresponding to the multiple sample images based on the hash codes of the multiple sample images and the hash codes of the class centers corresponding to the multiple sample images.

[0100] In this embodiment of the present application, each sample image corresponds to a class center. The server obtains first information between the two hash codes based on the hash code of the sample image and the hash code of the corresponding class center. Optionally, the server invokes an image retrieval model to process the class centers corresponding to multiple sample images to obtain predicted codes for the multiple class centers, and binarizes the predicted codes for the multiple class centers to obtain hash codes for the multiple class centers.

[0101] It should be noted that the embodiments of this application provide two representations to reflect the distance between any two hash codes in Hamming space. The first representation is the L2 norm ||·||. The smaller the L2 norm obtained based on any two hash codes, the smaller the distance between the two hash codes in Hamming space. The second representation is the Hamming distance. The smaller the Hamming distance obtained based on any two hash codes, the smaller the distance between the two hash codes in Hamming space.

[0102] Optionally, the first information between the plurality of sample images and the class centers corresponding to the plurality of sample images is expressed in the form of L2 norm. For example, the hash code of a sample image is represented as b i , the hash code of the class center corresponding to the sample image is represented by μ j , then the first information between the sample image and the class center is expressed as ||b i -μ j ||2. It should be understood that ||b i -μ j The smaller the value of ||2, the smaller the distance between the sample image and the center of the class in the Hamming space.

[0103] 405. The server obtains second information between the class centers based on the hash codes of the class centers.

[0104] In the embodiment of the present application, the server obtains the class centers corresponding to the plurality of sample images through the above step 403, and obtains the second information between any two class centers based on the hash codes of the two hash codes.

[0105] Optionally, the second information between the cluster centers is expressed as Hamming distance. For example, given any two hash codes b i and b j , the Hamming distance between them is expressed as Then, for any two class center hash codes μ p and μ q , the Hamming distance between them is expressed as It should be understood that the dist H (μ p , μ q ), the smaller the value of , the smaller the distance between the two cluster centers in the Hamming space.

[0106] 406. The server obtains a conditional probability between each cluster center based on the second information between each cluster center. The conditional probability is used to indicate the degree of dissimilarity between each cluster center.

[0107] In an embodiment of the present application, the conditional probability between each class center can also be used to indicate the Hamming distance between each class center. That is, the greater the conditional probability between each class center, the more dissimilar the two class centers are, and accordingly, the greater the Hamming distance between the two class centers.

[0108] Optionally, the server obtains the conditional probability between each class center based on the second information between each class center and the inter-class indicator matrix, where the inter-class indicator matrix is ​​used to ensure that the conditional probability between two identical class centers is zero.

[0109] The following is an explanation of the inter-class indicator matrix: A is a symmetric matrix with all diagonal elements being 0 and all other elements being 1. A is defined as follows: If p≠q, then A pq =1, otherwise A pq = 0, where p and q are used to represent any two class centers. It should be understood that the class centers corresponding to multiple sample images may overlap. By introducing the inter-class indicator matrix A, the conditional probability between two identical class centers is ensured to be 0, so that the server only calculates the conditional probability between two different class centers, thereby effectively reducing the computational complexity of model training.

[0110] Alternatively, for any two cluster centers μ p and μ q , the Hamming distance between the two cluster centers is expressed as Then the conditional probability p(A pq |M) Please refer to the following formula (1):

[0111]

[0112] Where, σ(x)=1 / (1+e -x ) is the sigmoid (S-type) function, in, The smaller the p(A pq |M) is larger, μ p and μ q The more dissimilar they are, the more μ p and μ q The larger the Hamming distance between them.

[0113] It should be noted that in the embodiments of the present application, the server executes steps 404 to 406 in the order described above. In some embodiments, the server first executes step 406, and then executes steps 405 and 404. In other embodiments, the server executes steps 404 to 406 simultaneously. The embodiments of the present application do not limit the order in which steps 404 to 406 are executed.

[0114] 407. The server calculates a loss value based on the first information between the multiple sample images and the class centers corresponding to the multiple sample images, the conditional probabilities between the class centers, and the quantization errors between the prediction codes and hash codes of the multiple sample images.

[0115] In an embodiment of the present application, a quantization error is used to indicate the error between a prediction code and a hash code, and is calculated based on the difference between the prediction code and the hash code of a plurality of sample images. The server can reduce the error of the image retrieval model by minimizing the value of the quantization error to improve the accuracy of the image retrieval model. The server calculates the loss value based on the target loss function, the first information between the plurality of sample images and the class centers corresponding to the plurality of sample images, the conditional probability between each class center, and the quantization error between the prediction code and the hash code of the plurality of sample images. The optimization goal of the target loss function is to minimize the distance between the plurality of sample images and the corresponding class centers in the Hamming space, maximize the distance between each class center in the Hamming space, and minimize the quantization error between the prediction code and the hash code of the sample image.

[0116] Optionally, the objective loss function L can be expressed as follows:

[0117]

[0118] Where Θ is the network parameter of the image retrieval model; M is the class center; L1 is the first loss function, and the optimization goal of L1 is to minimize the distance between multiple sample images and the class centers corresponding to the multiple sample images in the Hamming space; L2 is the second loss function, and the optimization goal of L2 is to maximize the distance between each class center in the Hamming space; γ is the balance coefficient used to weigh L1 and L2; represents the quantization error between the predicted code and the hash code; ν is the quantization loss coefficient, which is used to control the weight of the quantization error. Optionally, γ and ν are both hyperparameters. For example, γ is set to 0.1 and ν is set to 0.01. This embodiment of the present application is not limited to this.

[0119] It should be noted that, in some embodiments, the target loss function L does not include the quantization error between the prediction code and the hash code. That is, the server obtains the loss value based on the first information between the multiple sample images and the class centers corresponding to the multiple sample images, as well as the conditional probabilities between the class centers. This embodiment of the present application is not limited to this.

[0120] The following refers to formula (3) to explain L1:

[0121]

[0122] Where, σ 2is the intra-class variance, which is used to control the intra-class error between sample images of the same category. Optionally, σ 2 is a hyperparameter, for example, 2 It is set to 4, which is not limited in the present embodiment. From formula (3), it can be concluded that for any sample image, the smaller the distance between the sample image and the corresponding class center in the Hamming space, the smaller the distance between ||h i -μ j The smaller ||2 is, the smaller the L1 loss is. That is, the server can minimize the distance between multiple sample images and their corresponding class centers in the Hamming space by minimizing the L1 loss.

[0123] The following refers to formula (4) to explain L2:

[0124]

[0125] Where p and q represent any two cluster centers, φ pq It is based on the Φ in the above formula (1) pq Deformed, φ pq =0.5lcos(μ p , μ q ). Based on the above formula (1) and formula (4), it can be concluded that for any two class centers, the greater the distance between the two class centers in the Hamming space, the smaller φ pq The smaller it is, the smaller the L2 loss value is. In other words, the server can maximize the distance between the centers of each class in the Hamming space by minimizing the L2 loss value.

[0126] The theoretical derivation process of formula (4) is explained below in combination with the above formula (1) and the following formulas (5) to (7).

[0127] First, refer to formula (5):

[0128]

[0129] In the formula, p(M|A) represents the probability of the degree of dissimilarity between any two class centers given the inter-class indicator matrix, which is related to the second information between the class centers, and p(M) represents the probability of the class centers. Combined with the above formula (1), it can be concluded that by maximizing the posterior estimate on the right side of the formula in the above formula (5), the distance between the class centers in the Hamming space can be maximized. It should be understood that here, the distance between the class centers in the Hamming space is maximized by maximizing the above formula (1).

[0130] Second, combined with the above formula (1), considering that the class center M is a continuous variable in actual optimization, then The continuous estimation under discrete constraints refers to the following formula (6):

[0131]

[0132] Based on the above formula (6), the above formula (1) Replace with The following formula (7) is obtained:

[0133]

[0134] Third, maximize p(A pq |M) can be simplified to the above formula (4).

[0135] It should be noted that, after the above step 407, the server calculates the loss value according to a target loss function based on a dual constraint of class centers, which constrains the distance between sample images of the same category and their corresponding class centers, and constrains the distance between class centers of different categories, thereby avoiding the hash code overlap phenomenon caused by the excessive length of the hash code; further, the quantization error between the predicted codes and hash codes of multiple sample images is introduced, and the discrete output results of the image retrieval model are relaxed to the continuous space, thereby solving the discrete optimization problem of the image retrieval model and thereby improving the discriminability of the hash code generated by the image retrieval model.

[0136] 408. The server updates the network parameters and each class center of the image retrieval model based on the loss value to obtain an updated image retrieval model and each class center.

[0137] In an embodiment of the present application, the server calls the back propagation algorithm based on the loss value and the target loss function to update the network parameters of the image retrieval model and each class center to obtain an updated image retrieval model and each class center.

[0138] Optionally, taking the target loss function shown in formula (2) above as an example, the server calculates the target loss function L for h i and μ j The derivative of is the following formula (8), and then based on the loss value, the network parameters Θ and each class center M of the image retrieval model are updated. Formula (8) is as follows:

[0139]

[0140] It should be noted that after the server has gone through steps 401 to 408, it has obtained an updated image retrieval model and an updated class center. The server then performs the next iterative training based on the updated image retrieval model and the updated class center. The specific training process is similar to steps 401 to 408, with the difference being that, in steps 401 and 403, the server performs the relevant steps based on the initialized image retrieval model and the initialized class center, respectively. In the next iterative training process, the server performs the relevant steps based on the updated image retrieval model and the updated class center, respectively. Through this multiple iterative training method, the network parameters and class center of the image retrieval model are iteratively updated until the target training conditions are met, thereby obtaining a trained image retrieval model.

[0141] In addition, it is worth noting that in the embodiments of the present application, in addition to the network parameters of the image retrieval model, the class center is also regarded as a learnable parameter. This method of updating the class center according to the loss value of the image retrieval model during the training process can timely optimize the class center corresponding to the training data set to reduce the error of the image retrieval model, thereby effectively improving the training efficiency of the image retrieval model.

[0142] Moreover, after obtaining the trained image retrieval model, the server can input the image to be retrieved into the image retrieval model and perform binarization on the output of the image retrieval model to obtain the hash code of the image to be retrieved. Schematically, the image retrieval model is represented as f(·)=f(Θ;x i ), for any image input into the image retrieval model, the hash code of the image can be obtained by the following formula (9).

[0143] b i =sgn(f(Θ;x i )) (9)

[0144] Based on formula (9), it can be seen that although the training method of the image retrieval model provided in this application uses the class centers corresponding to the training dataset during the training process, the trained image retrieval model does not rely on the class centers corresponding to the training dataset when generating hash codes. Therefore, in the embodiments of this application, the trained image retrieval model can generate a corresponding hash code based on any image.

[0145] The present application provides a method for training an image retrieval model. After obtaining hash codes for multiple sample images through the image retrieval model, the class centers corresponding to the multiple image categories to which the multiple sample images belong are obtained. The image retrieval model and each class center are then trained based on the distance between each sample image and its corresponding class center, as well as the distance between each class center. This training method can reduce the computational complexity of model training, thereby improving the training efficiency of the model. Moreover, the trained image retrieval model can generate more discriminative hash codes, thereby effectively improving the accuracy of the image retrieval model while improving training efficiency.

[0146] In the above Figure 4 In the embodiment shown, the optimization objectives of the image retrieval model are to minimize the distances between multiple sample images and corresponding class centers in the Hamming space, maximize the distances between each class center in the Hamming space, and minimize the quantization error between the predicted code and the hash code of the sample image. Figure 5 Another specific implementation of the image retrieval model training method provided in this application is described. Figure 5 is a flowchart of another image retrieval model training method provided in an embodiment of the present application, such as Figure 5 As shown, the training method of the image retrieval model is applied to the server. In the embodiment of the present application, an optimization objective related to the regression matrix is ​​added to train the image retrieval model. The following is an example of the server performing the first iteration of training. The method includes the following steps:

[0147] 501. The server calls an image retrieval model to process multiple sample images to obtain prediction codes of the multiple sample images.

[0148] 502. The server performs binarization processing on the predicted codes of the multiple sample images to obtain hash codes of the multiple sample images.

[0149] 503. The server obtains class centers of multiple image categories based on the multiple image categories to which the multiple sample images belong.

[0150] In the embodiment of the present application, the implementation methods and optional execution order of the above steps 501 to 503 are the same as those of the above steps 401 to 403, so they are not repeated here.

[0151] 504. The server obtains a regression matrix based on the label matrix and the hash codes of the multiple sample images.

[0152] In the embodiment of the present application, the regression matrix is ​​used to provide a connection relationship between the label matrix and the hash codes of multiple sample images. Schematically, the regression matrix is ​​represented as

[0153] Optionally, the server calculates the regression matrix W based on the following formula (10).

[0154] W=(YY T ) -1 YB T (10)

[0155] Where Y is the label matrix; B is the hash code of multiple sample images.

[0156] The theoretical derivation process of the above formula (10) is explained below based on the following formulas (11) to (14).

[0157] In the embodiment of the present application, a regression matrix is ​​introduced, and a third loss function is added on the basis of the above formula (2). Schematically, the target loss function L of the image retrieval model is shown in the following formula (11):

[0158]

[0159] Where L3 is the third loss function, that is, compared with the above Figure 4 The target loss function in the embodiment shown in the figure has an additional optimization goal of improving the discriminability of the hash code generated by the image retrieval model. Wherein, L3 refers to the following formula (12):

[0160]

[0161] Where W T y i Used to represent b i The category coding of the corresponding image category, according to the above formula (12), can be known that by minimizing the loss value of L3, the similarity between the hash codes of sample images in the same image category can be maximized, that is, the hash codes of sample images of the same image category are made as similar as possible (in layman's terms, it is to encourage the image retrieval model to output the same binary code based on sample images in the same image category). In other words, the optimization goal of L3 is to reduce the changes between the hash codes of sample images in the same image category to increase the compactness between the hash codes of sample images in the same image category, thereby improving the discriminability of the hash codes generated by the image retrieval model, so that the server can determine the correct category of the image based on the generated hash code.

[0162] In addition, according to the above formula (11), in the embodiment of the present application, the target loss function L includes four variables, namely, the network parameter Θ of the image retrieval model, the class center M, the hash code B of multiple sample images, and the regression matrix W. The present application iteratively updates these four variables based on the alternating minimization algorithm. During the alternating minimization process, when one variable is updated, the other variables remain unchanged.

[0163] In this case, based on the alternating minimization algorithm, the server first keeps the network parameters Θ, the class center M, and the hash codes B of the image retrieval model unchanged, and then updates the regression matrix W. Based on this, the target loss function shown in the above formula (11) is simplified to obtain the following formula (13):

[0164]

[0165] Since formula (13) is a standard linear regression problem, after removing the irrelevant terms in formula (13), the following formula (14) is obtained:

[0166]

[0167] By calculating the gradient of formula (14) when the regression matrix W is 0, the closed-form solution of the regression matrix W can be obtained, which is the above formula (10).

[0168] It should be noted that in step 504, the implementation method of the server obtaining the regression matrix is ​​the first step of the server iteratively updating the four variables in the target loss function based on the alternating minimization algorithm, that is, keeping the network parameters Θ, class center M and hash codes B of the image retrieval model unchanged, and updating the regression matrix W, which can also be understood as determining the first variable in the target loss function.

[0169] 505. The server updates the hash codes of the multiple sample images based on the predicted codes of the multiple sample images, the label matrix, and the regression matrix to obtain updated hash codes of the multiple sample images.

[0170] In an embodiment of the present application, the server updates the hash codes of the multiple sample images based on the predicted codes of the multiple sample images, the label matrix, and the regression matrix determined according to the above step 504 to obtain updated hash codes of the multiple sample images.

[0171] Optionally, the server updates the hash codes of the multiple sample images based on the following formula (15):

[0172] B=sgn(W T Y+vH) -1 YB T (15)

[0173] The theoretical derivation process of the above formula (15) is explained below based on the following formulas (16) and (17).

[0174] According to the theoretical derivation of step 504 and formula (10) in step 504, the server first keeps the network parameters Θ, class center M, and hash codes B of the image retrieval model unchanged based on the alternating minimization algorithm, and updates the regression matrix W. Next, the server keeps the network parameters Θ, class center M, and regression matrix W of the image retrieval model unchanged and updates the hash codes B of the multiple sample images. Based on this, the target loss function shown in formula (11) is simplified to obtain the following formula (16):

[0175]

[0176] Given that Tr(B T B) is a constant, the above formula (16) is equivalent to the following formula (17):

[0177]

[0178] Based on this formula (17), the above formula (15) can be obtained.

[0179] It should be noted that in step 505, the implementation method of the server obtaining the updated hash codes of the multiple sample images is the second step of the server iteratively updating the four variables in the target loss function based on the alternating minimization algorithm, that is, keeping the network parameters Θ, class center M and regression matrix W of the image retrieval model unchanged, and updating the hash codes B of the multiple sample images, which can also be understood as determining the second variable in the target loss function.

[0180] 506. The server determines first information between the multiple sample images and the cluster centers corresponding to the multiple sample images based on the updated hash codes of the multiple sample images.

[0181] In the embodiment of the present application, the implementation of this step 506 is similar to the above-mentioned step 404, except that in step 404, the server uses the hash codes of multiple sample images, while in this step 506, the server uses the updated hash codes of multiple sample images, so this application will not repeat them here.

[0182] 507. The server obtains second information between the class centers based on the hash codes of the class centers.

[0183] 508. The server obtains a conditional probability between each cluster center based on the second information between each cluster center. The conditional probability is used to indicate the degree of dissimilarity between each cluster center.

[0184] It should be noted that the above steps 507 and 508 are the same as the above steps 405 and 406. In addition, the optional implementation method of the execution order of the above steps 506 to 508 is the same as the above steps 404 to 406, so this application will not repeat them here.

[0185] 509. The server obtains third information between the multiple sample images and the image categories corresponding to the multiple sample images based on the updated hash codes, label matrix, and regression matrix of the multiple sample images.

[0186] In the embodiment of the present application, the third information is used to indicate the error between the hash code of any sample image and the category code of the corresponding image category. Optionally, the expression of the third information refers to the above formula (12), that is, for any sample image, the third information between the sample image and the corresponding image category is expressed as It should be understood that The smaller the value of , the smaller the error between the hash code of the sample image and the category code of the corresponding image category, and the higher the discriminability of the hash code of the sample image.

[0187] It should be noted that, in some embodiments, the server first executes step 509 and then executes the above steps 506 to 508; in other embodiments, the server executes the above steps 506 to 509 synchronously, which is not limited in the embodiments of the present application.

[0188] 510. The server calculates a loss value based on first information between multiple sample images and class centers corresponding to the multiple sample images, conditional probabilities between the class centers, third information between multiple sample images and image categories corresponding to the multiple sample images, and quantization errors between predicted codes and hash codes of the multiple sample images.

[0189] In an embodiment of the present application, the server calculates a loss value based on a target loss function, first information between a plurality of sample images and the class centers corresponding to the plurality of sample images, conditional probabilities between the class centers, third information between the plurality of sample images and the image categories corresponding to the plurality of sample images, and quantization errors between the predicted codes and hash codes of the plurality of sample images. For example, the target loss function refers to the above formula (11).

[0190] It should be noted that, in some embodiments, the target loss function L does not include the quantization error between the prediction code and the hash code. That is, the server calculates the loss value based on the first information between the multiple sample images and the class centers corresponding to the multiple sample images, the conditional probabilities between the class centers, and the third information between the multiple sample images and the image categories corresponding to the multiple sample images. This embodiment of the present application is not limited to this.

[0191] 511. Based on the loss value, the server updates the network parameters of the image retrieval model and each class center to obtain an updated image retrieval model and each class center.

[0192] In the embodiment of the present application, the implementation of step 511 is the same as that of step 408 above, that is, the server calls the backpropagation algorithm based on the loss value and the target loss function to update the network parameters of the image retrieval model and each class center, thereby obtaining an updated image retrieval model and each class center. This application will not repeat this description.

[0193] The first point that needs to be explained is that in step 511, the server updates the network parameters of each class center and the image retrieval model based on the loss value, which is the third step of the server updating the four variables in the target loss function based on the alternating minimization algorithm, that is, given the regression matrix W and the hash code B of the updated multiple sample images, the network parameters Θ and the class center M of the image retrieval model are updated, which can also be understood as determining the remaining two variables in the target loss function.

[0194] The second point to note is that after the server completes steps 501 to 511, it obtains an updated image retrieval model and updated class centers. The server then performs the next round of iterative training based on the updated image retrieval model and updated class centers. Through this multiple rounds of iterative training, the network parameters and class centers of the image retrieval model are iteratively updated until the target training conditions are met, resulting in a trained image retrieval model.

[0195] Optionally, the following is an example of a method in which the server updates the four variables in the target loss function based on the alternating minimization algorithm to schematically illustrate the above steps 504 to 511, including but not limited to the following steps 1 to 4:

[0196] Step 1: The server obtains the regression matrix based on the label matrix and the hash codes of multiple sample images.

[0197] In step 1, the server maintains the network parameters of the image retrieval model, the centers of each class, and the hash codes of multiple sample images unchanged and calculates the regression matrix, that is, updates the regression matrix. The specific method of obtaining the regression matrix is ​​referred to in step 504 above and will not be repeated here.

[0198] Step 2: The server keeps the regression matrix, the network parameters of the image retrieval model, and each of the class centers unchanged, and updates the hash codes of the multiple sample images to obtain updated hash codes of the multiple sample images.

[0199] In step 2, the server updates the hash codes of the multiple sample images in a manner similar to step 505 above, which will not be described in detail in this application.

[0200] Step 3: The server keeps the regression matrix and the updated hash codes of the multiple sample images unchanged, and calculates the loss value based on the first information between the multiple sample images and the class centers corresponding to the multiple sample images, the second information between each of the class centers, and the regression matrix.

[0201] In step three, the server maintains the regression matrix and the updated hash codes of the multiple sample images unchanged, obtains third information between the multiple sample images and the image categories corresponding to the multiple sample images, and then calculates the loss value based on the first information between the multiple sample images and the class centers corresponding to the multiple sample images, the second information between each of the class centers, and the third information between the multiple sample images and the image categories corresponding to the multiple sample images. The specific method for obtaining the first information, the second information, and the third information is referred to above in steps 506 to 509, and the method for calculating the loss value is referred to above in step 510, which will not be repeated in this application.

[0202] Step 4: Based on the loss value, the server updates the network parameters of the image retrieval model and each of the centers of this type to obtain an updated image retrieval model and each of the centers of this type.

[0203] In step 4, the server updates the network parameters of the image retrieval model and each center of this type in a manner similar to step 511 above, which will not be repeated in this application.

[0204] See Table 1 below. Table 1 is a summary of the training process of the image retrieval model, that is, a summary of the above steps 501 to 511.

[0205] Schematically, when training the image retrieval model, the input information obtained by the server includes: training dataset X, label matrix Y, image retrieval model f(·), hash code length l and batch sampling size m. Parameters include intra-class variance σ 2, balance coefficient γ and quantization loss coefficient ν. The output information of the server includes: the network parameters Θ of the learned image retrieval model and the class center M corresponding to the training dataset.

[0206] As shown in Table 1, before starting training, the server performs an initialization process to generate the network parameters Θ and class centers M of the image retrieval model. The server then repeats the training process shown in Table 1 based on the training cycle and number of iterations until convergence. It should be noted that the training process shown in Table 1 has been described in detail in steps 501 to 511 above and will not be repeated here.

[0207] Table 1

[0208]

[0209] Reference below Figure 6 Taking the training method of the image retrieval model provided in the above steps 501 to 511 as an example, the training architecture of the image retrieval model provided in the embodiment of the present application is explained. Figure 6 This is a diagram of the training architecture of an image retrieval model provided in an embodiment of the present application. Figure 6 As shown in Figure 1, the training architecture of the image retrieval model consists of the following three parts:

[0210] The first part is the feature learning part. The server inputs the sample image into the image retrieval model, passes it through the second fully connected layer (also known as the fully connected hash layer), obtains the output result H, and passes the output result H to the second and third parts. The network structure of the image retrieval model can be found in Figure 2 .

[0211] The second part is based on a dual-constraint loss for the class center M. This dual-constraint loss constrains both the distances between sample images of the same class and their corresponding class centers and the distances between class centers of different classes. In other words, while minimizing the distances between sample images of the same class and their corresponding class centers, it maximizes the distances between class centers of different classes. Based on the output H of the first part and the dual-constraint loss from the second part, the server updates the class center M and passes the updated M to the first part.

[0212] The third part is the regression part. This regression part generates a more discriminative hash code B based on the label matrix Y of the training dataset and the output result H of the first part by introducing the regression matrix W, and passes the hash code B to the first part.

[0213] Next, we will continue to use the training method of the image retrieval model provided in steps 501 to 511 as an example to illustrate the beneficial effects of the embodiment of the present application based on the experimental results of the embodiment of the present application and related solutions. It should be noted that, for the sake of convenience, the training method of the image retrieval model provided in the embodiment of the present application is named "Deep Center-Based Dual-Constrained Hashing (DCDH)".

[0214] Before describing the experimental results of the embodiments of the present application and related solutions, the experimental solution of the present application is first introduced, which mainly includes the following four points:

[0215] First, standard data set.

[0216] Among them, standard datasets include YouTube Faces, FaceScrub, and VGGFace2.

[0217] YouTube Faces contains 3435 videos of 1595 characters, with each video having between 48 and 6070 frames. For each character, this application selects 40 images for training and 5 images for testing. Each sample image is 32×32 pixels in size.

[0218] FaceScrub contains 106,863 facial images of 530 characters. Similarly, this application randomly selects 5 images for each character as test images, and the remaining images are used as sample images to form the training dataset. The size of each sample image is 32×32, and the unit is pixels.

[0219] VGGFace2 contains 3,310,000 facial images of 9,131 people, and 8,631 of these people are divided into a training dataset. This application randomly selects 50 images of each public figure from the training dataset as test images, and the remaining images are used as sample images to obtain the training dataset. This application sets up two experimental methods based on VGGFace2, and therefore adjusts the size of each sample image to 32×32 and 160×160, respectively, in pixels.

[0220] Second, relevant plans.

[0221] Among them, the solutions that need to be compared with this application include the following three categories:

[0222] The first category is the schemes based on bigram / triplet similarity, including Deep Pairwise Supervised Hashing (DPSH), Deep Supervised Hashing with Triplet Labels (DTSH), Deep Supervised Discrete Hashing (DSDH), and Deep Class-Wise Hashing (DCWH).

[0223] The second category is deep hashing face retrieval schemes based on softmax classification loss, including Discriminative Deep Hashing (DDH), Discriminative DeepQuantization Hashing (DDQH), and Discriminative DeepAttention-Aware Hashing (DDAH).

[0224] The third category is the feature extraction scheme of supervised hash learning combined with pre-trained deep CNN, including the scheme of supervised discrete hashing (SDH) combined with CNN (expressed as SDH+CNN) and the scheme of fast supervised discrete hashing (FSDH) combined with CNN (expressed as FSDH+CNN).

[0225] It should be noted that each of the solutions in the second category uses its own network backbone. To more fairly compare their performance differences with this application, the DDQH solution in the second category is named DDQH*, which uses the same network backbone as this application. At the same time, the solutions in the first and third categories above all use the network backbone of this application for training.

[0226] The third point is the evaluation criteria.

[0227] Among them, when comparing the experimental results of this application with those of related solutions, the following five evaluation criteria are used for evaluation: Mean Average Precision (MAP); Precision of the top k returned samples (Top-k), where k≥1; Precision under different hash code lengths; Precision-Recall Curve; Recall under different hash code lengths.

[0228] Fourth, experimental parameters.

[0229] In the training process, the Adam algorithm is used to optimize the image retrieval model. At the same time, the network parameters Θ and the initial learning rate of the class center M of the image retrieval model are set to 5e-3, the total training cycle is set to 600 generations, the batch sampling size is set to 256, and the intra-class variance σ of the image retrieval model is set by cross-validation. 2 =4; balance coefficient γ = 0.1; quantization loss coefficient ν = 0.01. It should be noted that the experimental parameters here are only illustrative and are only used to compare this application with related solutions. They do not constitute a limitation on the various parameters in the above-mentioned embodiments of this application. In actual applications, developers can set the experimental parameters according to their needs, and this application does not impose any restrictions on this.

[0230] The following first describes the experimental results of this application and related solutions based on the two datasets of YouTube Faces and FaceScrub.

[0231] First, please refer to Table 2, which shows the experimental results of MAP comparison under different hash code lengths obtained based on the YouTube Faces and FaceScrub datasets. As shown in Table 2, the experimental results show that compared with other related schemes, the DCDH provided in this application achieves better MAP under all hash code lengths on the YouTube Faces and FaceScrub datasets.

[0232] On the YouTube Faces dataset, the DCDH proposed in this application improved the MAP at a 12-bit hash code length to 91.61%, nearly four percentage points ahead of the second-place candidate. At the other three hash code lengths, DCDH's MAP was still superior to other related solutions. On the FaceScrub dataset, the DCDH proposed in this application significantly outperformed other related solutions by at least five percentage points at all hash code lengths.

[0233] It should be noted that, with a 12-bit hash code length, the DCDH proposed in this application outperforms the second-place SDH+CNN by 20 percentage points. In comparison, the MAPs obtained by DPSH and DTSH based on these two datasets are relatively low, mainly because these two schemes cannot learn all sample information during training. The DCDH proposed in this application trains the image retrieval model based on class centers and quantization errors. This not only learns all sample information, but also reduces the computational complexity of model training, improving model training efficiency. Moreover, the trained image retrieval model can generate more discriminative hash codes, thereby effectively improving the MAP.

[0234] Table 2

[0235]

[0236] Next, please refer to Table 3, which shows the comparative experimental results of the top-5 accuracy of the first five returned samples under different hash code lengths based on the YouTube Faces and FaceScrub datasets. As shown in Table 3, the experimental results show that compared with related solutions, the DCDH provided by this application achieves better top-5 return accuracy on both the YouTube Faces and FaceScrub datasets than other related solutions, similar to the analysis results in Table 2 above.

[0237] Table 3

[0238]

[0239] Then, please refer to Figure 7 and Figure 8 , the precision (Precision), precision-recall curve (Precision-Recall Curve) and recall rate (Recall) results under different hash code lengths obtained by this application and related solutions on the two datasets of YouTube Faces and FaceScrub are described.

[0240] Figure 7 This is a schematic diagram of the experimental results of each solution provided in the embodiments of the present application, which is used to represent the experimental results of the present application and related solutions on the YouTube Faces dataset. Figure 7 Figure (a) shows the return accuracy curve within two Hamming distances corresponding to different hash code lengths; Figure 7 The middle (b) figure shows the precision-recall curve under the 48-bit hash code length; Figure 7 The middle figure (c) shows the recall rate curve under different hash code lengths.

[0241] Figure 8 This is a schematic diagram of the experimental results of another embodiment of the present application, which is used to represent the experimental results of the present application and related solutions on the FaceScrub dataset. Figure 8 Figure (a) shows the return accuracy curve within two Hamming distances corresponding to different hash code lengths; Figure 8 The middle (b) figure shows the precision-recall curve under the 48-bit hash code length; Figure 8 The middle figure (c) shows the recall rate curve under different hash code lengths.

[0242] from Figure 7 and Figure 8 It can be concluded that the performance of the DCDH provided by this application is better than other related solutions under the above three evaluation criteria. On the one hand, the target loss function in the DCDH provided by this application takes into account the distance between the sample image and the class center, which is beneficial for capturing the global semantic similarity of the training data set and reducing the impact of outlier samples on training; on the other hand, the target loss function in the DCDH provided by this application also takes into account the distance between class centers, which can effectively increase the distance between different class centers, avoid the hash code overlap phenomenon caused by the increase in hash code length, and make the hash code generated based on the DCDH provided by this application a more discriminative and compact hash code.

[0243] The following describes the experimental results of this application and related solutions based on the VGGFace2 dataset.

[0244] First of all, unlike the YouTube Faces and FaceScrub datasets, this application provides two experimental methods when conducting experiments based on the VGGFace2 dataset. The first experimental method is the same as the previous experimental method on YouTube Faces and FaceScrub, that is, the network backbone of this application is used for training and testing. This experimental method is referred to as the original network. At this time, the size of the sample image is 32×32. The second experimental method is to deploy a model that has been pre-trained on the VGGFace2 dataset as an initialization network, and then fine-tune the initialization network to obtain a pre-trained network. Experiments are conducted on the basis of the pre-trained network. This experimental method is referred to as the pre-trained network. At this time, the size of the sample image is 160×160. Schematically, the network structure of the initialization network is Inception-ResNet, and the fully connected layer originally used for classification in the initialization network is replaced with the embodiment of this application. Figure 2 The second fully connected layer (which can also be understood as a fully connected hash layer) shown is a pre-trained network obtained, and experiments are performed based on the pre-trained network.

[0245] Next, please refer to Table 4, which shows the experimental results of two experimental methods based on the VGGFace2 dataset. As shown in Table 4, the experimental results show that compared with related solutions, the performance of the DCDH provided by this application is better than other related solutions in both the original network and pre-trained network experimental methods.

[0246] Table 4

[0247]

[0248] Then, please refer to Figure 9 , Figure 9 This is a schematic diagram of the experimental results of another embodiment of the present application, which is used to represent the experimental results of the present application and related solutions on the VGGFace2 dataset. Figure 9 Figure (a) shows the precision-recall curves of the five schemes shown in Table 4 under the 48-bit hash code length; Figure 9 The middle (b) figure shows the accuracy curves of the five schemes shown in Table 4 for different numbers of samples before returning under the 48-bit hash code length. Figure 9 It can be concluded that the performance of the DCDH provided in this application is significantly better than other related solutions.

[0249] In summary, the image retrieval model training method of the embodiment of the present application obtains the hash codes of multiple sample images through the image retrieval model, obtains the class centers corresponding to the multiple image categories to which the multiple sample images belong, and then trains the class centers of the image retrieval model based on the distance between each sample image and its corresponding class center and the distance between each class center. This training method can reduce the computational complexity of model training, thereby improving the training efficiency of the model. Moreover, the trained image retrieval model can generate more discriminative hash codes, thereby effectively improving the accuracy of the image retrieval model while improving training efficiency.

[0250] Schematically, the image retrieval model obtained by the training method of the image retrieval model provided in this application can be applied in scenarios including but not limited to a variety of image retrieval scenarios. For example, taking the social entertainment scenario as an example, an application running on a terminal is developed based on the image retrieval model, and the application provides a query function. For example, a user can perform a query operation on the terminal to upload a celebrity photo to be queried. In response to the user's query operation, the terminal calls the image retrieval model, and the image retrieval model outputs a hash code of the celebrity photo to be queried. The terminal searches the image database based on the hash code, obtains other photos of the corresponding celebrity, and displays them on the terminal screen, thereby realizing the "celebrity query" function. For another example, a user can perform a query operation on the terminal to upload a photo of the user himself. In response to the user's query operation, the terminal calls the image retrieval model, and the image retrieval model outputs a hash code of the photo of the user himself. The terminal searches the image database based on the hash code, obtains the celebrity photo with the highest similarity to the photo of the user himself, and displays it, thereby realizing the "most similar celebrity face" function.

[0251] Figure 10 This is a structural diagram of a training device for an image retrieval model provided in accordance with an embodiment of the present application. The device is used to execute the steps of the training method for the above-mentioned image retrieval model, see Figure 10 The device includes: a processing module 1001, a class center acquisition module 1002, a calculation module 1003 and a training module 1004.

[0252] A processing module 1001 is configured to process a plurality of sample images based on an image retrieval model to obtain hash codes of the plurality of sample images;

[0253] A class center acquisition module 1002 is configured to acquire class centers of the multiple image categories based on the multiple image categories to which the multiple sample images belong;

[0254] a calculation module 1003, configured to calculate a loss value based on first information between the plurality of sample images and the class centers corresponding to the plurality of sample images, and second information between the class centers, wherein the first information and the second information are respectively used to indicate distances between hash codes of the corresponding sample images in a Hamming space;

[0255] The training module 1004 is used to train the image retrieval model and each of the class centers based on the loss value.

[0256] In an optional implementation, the calculation module 1003 includes:

[0257] a first acquiring unit, configured to obtain first information between the plurality of sample images and the class centers corresponding to the plurality of sample images based on the hash codes of the plurality of sample images and the hash codes of the class centers corresponding to the plurality of sample images;

[0258] A second obtaining unit is configured to obtain second information between the centers of the same type based on the hash code of the centers of the same type;

[0259] A third obtaining unit is configured to obtain a conditional probability between the centers of the type based on the second information between the centers of the type, where the conditional probability indicates a degree of dissimilarity between the centers of the type;

[0260] A calculation unit is used to calculate the loss value based on the first information between the multiple sample images and the class centers corresponding to the multiple sample images and the conditional probability between each of the class centers.

[0261] In an optional implementation, the third acquiring unit is configured to:

[0262] Based on the second information between the centers of the class and the inter-class indicator matrix, the conditional probability between the centers of the class is obtained. The inter-class indicator matrix is ​​used to ensure that the conditional probability between two identical centers of the class is 0.

[0263] In an optional implementation, the processing module 1001 is configured to:

[0264] Calling the image retrieval model to process the multiple sample images to obtain predictive codes for the multiple sample images;

[0265] Binarizing the predicted codes of the multiple sample images to obtain hash codes of the multiple sample images;

[0266] The calculation module 1003 is used to:

[0267] The loss value is calculated based on the first information between the multiple sample images and the class centers corresponding to the multiple sample images, the second information between each of the class centers, and the quantization error between the predicted codes and hash codes of the multiple sample images, where the quantization error is calculated based on the difference between the predicted codes and hash codes of the multiple sample images.

[0268] In an optional implementation, the device further includes:

[0269] A regression matrix acquisition module, configured to acquire a regression matrix based on the label matrix and the hash codes of the plurality of sample images;

[0270] The calculation module 1003 is used to:

[0271] The loss value is calculated based on first information between the multiple sample images and the class centers corresponding to the multiple sample images, second information between the respective class centers, and the regression matrix.

[0272] In an optional implementation, the calculation module 1003 is configured to:

[0273] Obtaining, based on the hash codes of the multiple sample images, the label matrix, and the regression matrix, third information between the multiple sample images and the image categories corresponding to the multiple sample images, the third information being used to indicate an error between the hash code of any sample image and a category code of the corresponding image category;

[0274] The loss value is calculated based on first information between the multiple sample images and the class centers corresponding to the multiple sample images, second information between each of the class centers, and third information between the multiple sample images and the image categories corresponding to the multiple sample images.

[0275] In an optional implementation, the calculation module 1003 is configured to:

[0276] Keeping the regression matrix, the network parameters of the image retrieval model, and the centers of each class unchanged, the hash codes of the multiple sample images are updated to obtain updated hash codes of the multiple sample images;

[0277] Keeping the regression matrix and the updated hash codes of the multiple sample images unchanged, calculating the loss value based on the first information between the multiple sample images and the class centers corresponding to the multiple sample images, the second information between the class centers, and the regression matrix;

[0278] The training module is used to update the network parameters of the image retrieval model and each of the class centers based on the loss value to obtain an updated image retrieval model and each of the class centers.

[0279] The image retrieval model training device of the embodiment of the present application obtains the class centers corresponding to the multiple image categories to which the multiple sample images belong after obtaining hash codes for the multiple sample images through the image retrieval model, thereby training the image retrieval model and each class center based on the distance between each sample image and its corresponding class center and the distance between each class center. This training device can reduce the computational complexity of model training, thereby improving the training efficiency of the model. Moreover, the trained image retrieval model can generate more discriminative hash codes, thereby effectively improving the accuracy of the image retrieval model while improving training efficiency.

[0280] It should be noted that the image retrieval model training device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when training the image retrieval model. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image retrieval model training device provided in the above embodiment and the image retrieval model training method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0281] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory is used to store at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed in the training method of the image retrieval model in the embodiment of the present application.

[0282] In some embodiments, the computer program involved in the embodiments of the present application can be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected by a wired network or a wireless network. Multiple computer devices distributed at multiple locations and interconnected by a wired network or a wireless network can constitute a blockchain system.

[0283] Taking the computer device as a server as an example, Figure 11 This is a structural diagram of a server provided according to an embodiment of the present application. The server 1100 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1101 and one or more memories 1102, wherein the memory 1102 stores at least one computer program, and the at least one computer program is loaded and executed by the processor 1101 to implement the image retrieval model training method provided by the above-mentioned various method embodiments. Of course, the server can also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server can also include other components for implementing device functions, which will not be described here.

[0284] An embodiment of the present application also provides a computer-readable storage medium, which is applied to a computer device, and the computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the operations performed by the computer device in the training method of the image retrieval model of the above embodiment.

[0285] The present application also provides a computer program product or computer program, which includes computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the image retrieval model training method provided in the various optional implementations described above.

[0286] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0287] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A training method for an image retrieval model, characterized in that: The method comprises: Invoking an image retrieval model to process a plurality of sample images to obtain prediction codes of the plurality of sample images; performing binarization processing on the prediction codes of the plurality of sample images to obtain hash codes of the plurality of sample images; Based on the multiple image categories to which the multiple sample images belong, obtaining class centers of the multiple image categories; obtaining, based on the hash codes of the multiple sample images and the hash codes of the class centers corresponding to the multiple sample images, first information between the multiple sample images and the class centers corresponding to the multiple sample images, wherein the first information is used to indicate a distance between the hash code of any sample image and the class center corresponding to the sample image in a Hamming space; Based on the hash codes of the cluster centers, second information between the cluster centers is obtained, where the second information indicates the distance between the hash codes of any two cluster centers in the Hamming space; based on the second information between the cluster centers, conditional probabilities between the cluster centers are obtained, where the conditional probabilities indicate the degree of dissimilarity between the cluster centers; Obtaining a regression matrix based on a label matrix and hash codes of the plurality of sample images, wherein the label matrix is ​​used to indicate the image categories to which the plurality of sample images belong; obtaining third information between the plurality of sample images and the image categories corresponding to the plurality of sample images based on the hash codes of the plurality of sample images, the label matrix, and the regression matrix, wherein the third information is used to indicate an error between the hash code of any sample image and a category code of the corresponding image category; Calculating a loss value based on first information between the multiple sample images and class centers corresponding to the multiple sample images, conditional probabilities between the class centers, third information between the multiple sample images and the image categories corresponding to the multiple sample images, and quantization errors between predicted codes and hash codes of the multiple sample images, where the quantization errors are calculated based on differences between the predicted codes and hash codes of the multiple sample images; Based on the loss value, the image retrieval model and each of the class centers are trained.

2. The method according to claim 1, characterized in that The obtaining of the conditional probabilities between the cluster centers based on the second information between the cluster centers includes: Based on the second information between the class centers and an inter-class indicator matrix, a conditional probability between the class centers is obtained. The inter-class indicator matrix is ​​used to ensure that the conditional probability between two identical class centers is zero.

3. The method according to claim 1, characterized in that The calculating the loss value based on first information between the multiple sample images and the class centers corresponding to the multiple sample images, conditional probabilities between the class centers, third information between the multiple sample images and the image categories corresponding to the multiple sample images, and quantization errors between predicted codes and hash codes of the multiple sample images includes: Keeping the regression matrix, the network parameters of the image retrieval model, and each of the class centers unchanged, the hash codes of the multiple sample images are updated to obtain updated hash codes of the multiple sample images; Keeping the regression matrix and the updated hash codes of the multiple sample images unchanged, calculating the loss value based on the first information between the multiple sample images and the class centers corresponding to the multiple sample images, the second information between the class centers, and the regression matrix; The training of the image retrieval model and each of the class centers based on the loss value includes: Based on the loss value, the network parameters of the image retrieval model and each of the class centers are updated to obtain an updated image retrieval model and each of the class centers.

4. A training device for an image retrieval model, characterized in that: The device comprises: a processing module, configured to call an image retrieval model to process a plurality of sample images to obtain prediction codes of the plurality of sample images; and perform binarization processing on the prediction codes of the plurality of sample images to obtain hash codes of the plurality of sample images; a class center acquisition module, configured to acquire class centers of the multiple image categories based on the multiple image categories to which the multiple sample images belong; a calculation module, configured to obtain, based on the hash codes of the multiple sample images and the hash codes of the class centers corresponding to the multiple sample images, first information between the multiple sample images and the class centers corresponding to the multiple sample images, wherein the first information is used to indicate a distance between the hash code of any sample image and the class center corresponding to the sample image in a Hamming space; The calculation module is further configured to obtain, based on the hash codes of the cluster centers, second information between the cluster centers, the second information being used to indicate the distance between the hash codes of any two cluster centers in the Hamming space; and obtain, based on the second information between the cluster centers, conditional probabilities between the cluster centers, the conditional probabilities being used to indicate the degree of dissimilarity between the cluster centers. A regression matrix acquisition module, configured to acquire a regression matrix based on a label matrix and hash codes of the plurality of sample images, wherein the label matrix is ​​used to indicate the image categories to which the plurality of sample images belong; The calculation module is further configured to obtain, based on the hash codes of the multiple sample images, the label matrix, and the regression matrix, third information between the multiple sample images and the image categories corresponding to the multiple sample images, wherein the third information is configured to indicate an error between the hash code of any sample image and the category code of the corresponding image category; The calculation module is further configured to calculate a loss value based on first information between the multiple sample images and the class centers corresponding to the multiple sample images, conditional probabilities between the class centers, third information between the multiple sample images and the image categories corresponding to the multiple sample images, and a quantization error between the predicted codes and the hash codes of the multiple sample images, where the quantization error is calculated based on a difference between the predicted codes and the hash codes of the multiple sample images; A training module is used to train the image retrieval model and each of the class centers based on the loss value.

5. The device according to claim 4, characterized in that The calculation module is used for: Based on the second information between the class centers and an inter-class indicator matrix, a conditional probability between the class centers is obtained. The inter-class indicator matrix is ​​used to ensure that the conditional probability between two identical class centers is zero.

6. The device according to claim 4, characterized in that The calculation module is used for: Keeping the regression matrix, the network parameters of the image retrieval model, and each of the class centers unchanged, the hash codes of the multiple sample images are updated to obtain updated hash codes of the multiple sample images; Keeping the regression matrix and the updated hash codes of the multiple sample images unchanged, calculating the loss value based on the first information between the multiple sample images and the class centers corresponding to the multiple sample images, the second information between the class centers, and the regression matrix; The training module is used to: Based on the loss value, the network parameters of the image retrieval model and each of the class centers are updated to obtain an updated image retrieval model and each of the class centers.

7. A computer device, characterized in that: The computer device includes a processor and a memory, the memory is used to store at least one computer program, and the at least one computer program is loaded by the processor and executes the training method of the image retrieval model according to any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the training method of the image retrieval model according to any one of claims 1 to 3.

Images