Image retrieval method and apparatus, and storage medium and electronic device

Abstract

The present application relates to the technical field of computers. Disclosed are an image retrieval method and apparatus, and a storage medium and an electronic device. The method comprises: using a preset deep hashing model to process an image to be retrieved, so as to obtain an image hash code, wherein the preset deep hashing model is obtained by means of performing incremental learning training on the basis of a plurality of preset class centers and query data; and on the basis of a preset image corresponding to a preset hash code matching the image hash code, obtaining a matched image. The present application effectively improves the accuracy of image retrieval.

Description

Image retrieval method, device, storage medium and electronic device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on February 27, 2024, with application number 202410222358.2 and application name “Image Retrieval Method, Device, Storage Medium and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to an image retrieval method, device, storage medium and electronic device. Background Art

[0003] It will become increasingly important to provide users with fast and accurate image retrieval functions in massive image data, and to ensure that this function can adapt to the retrieval needs of the ever-increasing number of images.

[0004] The deep hash image retrieval method based on the deep hash model can input the query image input by the user into the deep hash model, extract the low-dimensional hash code, and perform similarity matching with the hash codes pre-extracted in the database, so as to quickly retrieve similar images. Technical issues

[0005] The current deep hashing model has the problem of overlapping class centers between old data (original data) and new data (incremental data) during the end-to-end incremental learning process. As a result, when users want to retrieve images of a certain category, they may retrieve images of different categories that overlap with their class centers, thereby reducing the image retrieval accuracy. Technical Solutions

[0006] The embodiment of the present application provides an image retrieval solution that can effectively improve image retrieval accuracy.

[0007] The embodiments of this application provide the following technical solutions:

[0008] According to one embodiment of the present application, an image retrieval method includes: obtaining an image to be retrieved; processing the image to be retrieved using a preset deep hash model to obtain an image hash code corresponding to the image to be retrieved, wherein the preset deep hash model is obtained by end-to-end incremental learning training based on multiple preset class centers and query data, and the multiple preset class centers are pre-constructed before model training. During the incremental learning training, the original image or incremental image in the query data is clustered to the preset class centers corresponding to its data category; matching the image hash code with a preset hash code in a hash code set to obtain a preset hash code that matches the image hash code; and obtaining a matching image of the image to be retrieved based on a preset image corresponding to the preset hash code that matches the image hash code.

[0009] In some embodiments of the present application, the preset deep hash model is trained in the following manner: class centers are pre-constructed before training the preset deep hash model to obtain multiple preset class centers, wherein each preset class center corresponds to a data category; training data is sampled from the original data set and the incremental data set to obtain query data; end-to-end incremental learning training is performed on the deep hash model to be trained based on the query data and the multiple preset class centers until a predetermined training end condition is met to obtain the trained preset deep hash model, wherein, during the incremental learning training process, the original data or incremental data in the query data are clustered to the preset class centers corresponding to their data categories.

[0010] In some embodiments of the present application, the class center is pre-constructed before the preset deep hash model is trained to obtain multiple preset class centers, including: using a Hadamard matrix to generate cluster centers to obtain multiple Hadamard cluster centers that are mutually orthogonal and bit-balanced; and obtaining the multiple preset class centers based on the multiple Hadamard cluster centers.

[0011] In some embodiments of the present application, the multiple preset class centers are obtained based on the multiple Hadamard cluster centers, including: if the number of the multiple Hadamard cluster centers is less than the number of data categories in the training data set, cluster centers are generated using Bernoulli distribution to obtain multiple mutually orthogonal and bit-balanced Bernoulli cluster centers, wherein a generation probability and constraint conditions are set for the Bernoulli distribution, the generation probability is one-half, and the constraint condition is the distance constraint value between the class center generated by the Bernoulli distribution and the Hadamard cluster center; the multiple preset class centers are obtained based on the multiple Hadamard cluster centers and the multiple Bernoulli cluster centers.

[0012] In some embodiments of the present application, the end-to-end incremental learning training of the deep hash model to be trained based on the query data and the multiple preset class centers is performed until a predetermined training end condition is met to obtain the trained preset deep hash model, including: inputting the sample image in the query data into the deep hash model to be trained, and obtaining a sample hash code of the sample image output by the deep hash model to be trained; if the sample image comes from the incremental data set, a first loss is calculated based on the sample hash code and the multiple preset class centers; if the sample image comes from the original data set, a second loss is calculated based on the sample hash code and the multiple preset class centers; according to the first loss or the second loss, the parameters of the deep hash model to be trained are adjusted until the predetermined training end condition is met to obtain the trained preset deep hash model.

[0013] In some embodiments of the present application, the first loss is calculated based on the sample hash code and the multiple preset class centers, including: calculating a center quantization loss based on the sample hash code and the preset class center corresponding to the data category of the sample image; calculating a similarity loss based on the sample hash code and the image feature data of the incremental image in the incremental data set; and fusing the center quantization loss and the similarity loss to obtain the first loss.

[0014] In some embodiments of the present application, the second loss is calculated based on the sample hash code and the multiple preset class centers, including: calculating a center quantization loss based on the sample hash code and the preset class center corresponding to the data category of the sample image; calculating a similarity loss based on the sample hash code and the image feature data of the incremental image in the incremental data set; calculating a memory loss based on the sample hash code and the image feature data of the original image in the original data set; and fusing the center quantization loss, the similarity loss and the memory loss to obtain the second loss.

[0015] According to one embodiment of the present application, an image retrieval device includes: an acquisition module for acquiring an image to be retrieved; a processing module for processing the image to be retrieved using a preset deep hash model to obtain an image hash code corresponding to the image to be retrieved, wherein the preset deep hash model is obtained by end-to-end incremental learning training based on multiple preset class centers and query data, and the multiple preset class centers are pre-constructed before model training. During the incremental learning training, the original image or incremental image in the query data is clustered to the preset class centers corresponding to its data category; a matching module for matching the image hash code with a preset hash code in a hash code set to obtain a preset hash code matched by the image hash code; and an output module for obtaining a matching image of the image to be retrieved based on a preset image corresponding to the preset hash code matched by the image hash code.

[0016] In some embodiments of the present application, the device also includes a training module, which is used to: pre-construct class centers before training the preset deep hash model to obtain multiple preset class centers, wherein each preset class center corresponds to a data category; sample training data from the original data set and the incremental data set to obtain query data; perform end-to-end incremental learning training on the deep hash model to be trained based on the query data and the multiple preset class centers until the predetermined training end conditions are met, thereby obtaining the trained preset deep hash model, wherein, during the incremental learning training process, the original data or incremental data in the query data are clustered to the preset class centers corresponding to their data categories.

[0017] In some embodiments of the present application, the training module is used to: generate cluster centers using a Hadamard matrix to obtain multiple Hadamard cluster centers that are mutually orthogonal and bit-balanced; and obtain the multiple preset cluster centers based on the multiple Hadamard cluster centers.

[0018] In some embodiments of the present application, the training module is used to: if the number of the multiple Hadamard cluster centers is less than the number of data categories in the training data set, generate cluster centers using the Bernoulli distribution method to obtain multiple mutually orthogonal and bit-balanced Bernoulli cluster centers, wherein the generation probability and constraint conditions are set for the Bernoulli distribution, the generation probability is one-half, and the constraint condition is the distance constraint value between the class center generated by the Bernoulli distribution and the Hadamard cluster center; based on the multiple Hadamard cluster centers and the multiple Bernoulli cluster centers, the multiple preset class centers are obtained.

[0019] In some embodiments of the present application, the training module is used to: input the sample image in the query data into the deep hash model to be trained, and obtain the sample hash code of the sample image output by the deep hash model to be trained; if the sample image comes from the incremental data set, a first loss is calculated based on the sample hash code and the multiple preset class centers; if the sample image comes from the original data set, a second loss is calculated based on the sample hash code and the multiple preset class centers; according to the first loss or the second loss, the parameters of the deep hash model to be trained are adjusted until the predetermined training end conditions are met, thereby obtaining the preset deep hash model after training.

[0020] In some embodiments of the present application, the training module is used to: calculate a central quantization loss based on the preset class center corresponding to the sample hash code and the data category of the sample image; calculate a similarity loss based on the sample hash code and the image feature data of the incremental image in the incremental data set; and fuse the central quantization loss and the similarity loss to obtain the first loss.

[0021] In some embodiments of the present application, the training module is used to: calculate a central quantization loss based on the sample hash code and the preset class center corresponding to the data category of the sample image; calculate a similarity loss based on the sample hash code and the image feature data of the incremental image in the incremental data set; calculate a memory loss based on the sample hash code and the image feature data of the original image in the original data set; and fuse the central quantization loss, the similarity loss and the memory loss to obtain the second loss.

[0022] According to another embodiment of the present application, a storage medium stores a computer program thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the method described in the embodiment of the present application.

[0023] According to another embodiment of the present application, an electronic device may include: a memory storing a computer program; and a processor reading the computer program stored in the memory to execute the method described in the embodiment of the present application.

[0024] According to another embodiment of the present application, a computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations described in the embodiments of the present application. Beneficial effects

[0025] In an embodiment of the present application, an image to be retrieved is obtained; the image to be retrieved is processed using a preset deep hash model to obtain an image hash code corresponding to the image to be retrieved, wherein the preset deep hash model is obtained by end-to-end incremental learning training based on multiple preset class centers and query data, and the multiple preset class centers are pre-constructed before model training. During the incremental learning training, the original image or incremental image in the query data is clustered to the preset class centers corresponding to its data category; the image hash code is matched with a preset hash code in a hash code set to obtain a preset hash code that matches the image hash code; and a matching image of the image to be retrieved is obtained according to a preset image corresponding to the preset hash code that matches the image hash code.

[0026] In this way, during image retrieval, a preset deep hash model is sampled to process the image to be retrieved to obtain an image hash code. The preset deep hash model is obtained by end-to-end incremental learning training based on multiple preset class centers and query data. Since multiple preset class centers are pre-built before model training, and during the incremental learning training process, the original image or incremental image in the query data is clustered to the preset class centers corresponding to its data category, it can effectively avoid the problem of class center overlap between old data (original image) and new data (incremental image) during the end-to-end incremental learning process of the deep hash model. Furthermore, the image hash code obtained based on the preset deep hash model can be accurately matched to the corresponding image retrieved, effectively improving the image retrieval accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0028] FIG1 shows a flowchart of an image retrieval method according to an embodiment of the present application.

[0029] FIG2 shows a structural block diagram of a deep hashing model according to an embodiment of the present application.

[0030] FIG3 shows a schematic diagram of concept drift according to an embodiment of the present application.

[0031] FIG4 shows a block diagram of an image retrieval device according to an embodiment of the present application.

[0032] FIG5 shows a block diagram of an electronic device according to an embodiment of the present application.

[0033] Implementation Methods of the Application

[0034] The present disclosure will be further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the examples provided herein are merely for explaining the present disclosure and are not intended to limit the present disclosure. In addition, the examples provided below are partial examples for implementing the present disclosure, rather than providing all examples for implementing the present disclosure. In the absence of conflict, the technical solutions described in the examples of the present disclosure may be implemented in any combination.

[0035] It should be noted that, in the embodiments of the present disclosure, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or apparatus comprising a series of elements includes not only the elements explicitly stated, but also other elements not explicitly listed, or also includes elements inherent to the implementation of the method or apparatus. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other related elements (such as steps in the method or units in the apparatus, for example, a unit may be part of a circuit, part of a processor, part of a program or software, etc.) in the method or apparatus comprising the element.

[0036] For example, the image retrieval method provided by the embodiment of the present disclosure includes a series of steps, but the image retrieval method provided by the embodiment of the present disclosure is not limited to the recorded steps. Similarly, the image retrieval device provided by the embodiment of the present disclosure includes a series of units, but the device provided by the embodiment of the present disclosure is not limited to including the units explicitly recorded, and may also include units that need to be set up to obtain relevant information or perform processing based on the information.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure pertains. The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure.

[0038] Figure 1 schematically illustrates a flow chart of an image retrieval method according to an embodiment of the present application. The image retrieval method may be executed by any device and / or server with processing capabilities, such as a television, computer, mobile phone, smartwatch, or home appliance, and a server such as a cloud server or a physical server.

[0039] As shown in FIG1 , the image retrieval method may include steps S110 to S140 .

[0040] Step S110, obtaining an image to be retrieved; step S120, using a preset deep hash model to process the image to be retrieved to obtain an image hash code corresponding to the image to be retrieved, wherein the preset deep hash model is obtained by end-to-end incremental learning training based on multiple preset class centers and query data, and the multiple preset class centers are pre-constructed before model training. During the incremental learning and training process, the original image or incremental image in the query data is clustered to the preset class centers corresponding to its data category; step S130, matching the image hash code with a preset hash code in a hash code set to obtain a preset hash code that matches the image hash code; step S140, obtaining a matching image of the image to be retrieved based on a preset image corresponding to the preset hash code that matches the image hash code.

[0041] The image to be retrieved is the query image used for retrieval. The image to be retrieved is input into a preset deep hash model, which processes the image to be retrieved to obtain an image hash code corresponding to the image to be retrieved.

[0042] The preset deep hash model can specifically be a model composed of a neural network. The neural network that constitutes the preset deep hash model can be selected based on different scenario requirements. For example, the EfficientNet network can be selected for large-scale retrieval systems, the MobileNet network can be selected for small devices such as mobile devices, or the AlexNet network can be selected based on requirements.

[0043] In one embodiment, referring to FIG2 , the neural network in the preset deep hash model may include a convolution and pooling layer 210, a fully connected layer 220, and a hash layer 230. The preset deep hash model processes the image to be retrieved. Specifically, the convolution and pooling layer 210 may first perform convolution and pooling on the image to be retrieved to obtain a first-layer feature. The fully connected layer 220 then performs a fully connected process on the first-layer feature to obtain a second-layer feature. Finally, the hash layer 230 performs quantization using a hash function (e.g., an sgn function) to obtain a hash code corresponding to the image (i.e., an image hash code).

[0044] Among them, the deep hash model in the related art is trained by generating cluster centers during the training process, and the training data of the model often grows dynamically over time. As shown in Figure 3, in the related art, when the distribution of old data (original data) in unit 310 in Figure 3 is significantly different from the distribution of new data (incremental data) in unit 320, the deep hash model will produce the concept drift 330 phenomenon shown in Figure 3 (due to the appearance of incremental data Class3, the class centers of the original data Class1 and Class2 appear to be offset and overlapped), thereby affecting the image retrieval performance of the static hashing method.

[0045] Different from the related art, in this application, the preset deep hash model is obtained by end-to-end incremental learning training based on multiple preset class centers and query data. The multiple preset class centers are pre-built before model training to ensure that there is a reasonable distance between each class center. During the incremental learning and training process, the original image (i.e., the image in the original data) or the incremental image (i.e., the image in the incremental data) in the query data is clustered to the preset class centers corresponding to its data category, which can effectively avoid the problem of class center overlap between old data (original data) and new data (incremental data) during the end-to-end incremental learning process of the deep hash model.

[0046] The image hash code is matched with a preset hash code in a preset hash code set to obtain a preset hash code for the image hash code match (for example, the similarity between the image hash code and the preset hash codes in the preset hash code set can be calculated, and the preset hash code with the greatest similarity is determined as the preset hash code for the image hash code match, where the similarity can be a similarity such as Hamming distance); the preset image corresponding to the preset hash code for the image hash code match can be used as the matching image of the image to be retrieved obtained by detection.

[0047] In this way, based on steps S110 to S140, during image retrieval, a preset deep hash model is sampled to process the image to be retrieved to obtain an image hash code. The preset deep hash model is obtained by end-to-end incremental learning training based on multiple preset class centers and query data. Since the multiple preset class centers are pre-built before model training, and the original data or incremental data in the query data are clustered to the preset class centers corresponding to their data categories during the incremental learning training process, the problem of class center overlap between old data (original data) and new data (incremental data) can be effectively avoided during the end-to-end incremental learning process of the deep hash model. Furthermore, the image hash code obtained by processing based on the preset deep hash model can be accurately matched to retrieve the corresponding image, effectively improving the image retrieval accuracy.

[0048] The following describes further optional specific embodiments of each step performed when performing image retrieval in the embodiment of Figure 1.

[0049] In one embodiment, the preset deep hash model can be specifically obtained by training in the following manner: class centers are pre-constructed before training the preset deep hash model to obtain multiple preset class centers, wherein each preset class center corresponds to a data category; training data is sampled from the original data set and the incremental data set to obtain query data; end-to-end incremental learning training is performed on the deep hash model to be trained based on the query data and the multiple preset class centers until the predetermined training end conditions are met, thereby obtaining the trained preset deep hash model, wherein, during the incremental learning training process, the original data or incremental data in the query data are clustered to the preset class centers corresponding to their data categories.

[0050] The deep hash model to be trained is a model that has been pre-trained based on the original dataset. After the incremental dataset is added, the deep hash model to be trained needs to be incrementally trained to obtain a preset deep hash model with retrieval capabilities for the incremental data.

[0051] Before training the deep hash model, the class center is pre-built to obtain multiple preset class centers, where each preset class center corresponds to a data category. For example, data categories such as Class1, Class2 and Class3 correspond to a preset class center respectively.

[0052] Based on the query data and multiple preset class centers, the deep hash model to be trained is trained end-to-end incrementally until the predetermined training end conditions are met (which can be set according to actual conditions, such as the retrieval accuracy is higher than a predetermined threshold or the number of learning times reaches a predetermined number, etc.), and the trained preset deep hash model is obtained.

[0053] Each preset class center corresponds to a data category, and during the incremental learning training process, the original data or incremental data in the query data are clustered to the preset class center corresponding to its data category. Since the multiple preset class centers are pre-built before model training, it can ensure that there is a reasonable distance between each class center. Furthermore, it can effectively avoid the problem of class center overlap between old data (original data) and new data (incremental data) during the end-to-end incremental learning process of the deep hash model.

[0054] Furthermore, in one embodiment, the pre-construction of class centers before training the preset deep hash model to obtain multiple preset class centers may include: using a Hadamard matrix to generate cluster centers to obtain multiple Hadamard cluster centers that are mutually orthogonal and bit-balanced; and obtaining the multiple preset class centers based on the multiple Hadamard cluster centers.

[0055] The Hadamard matrix is ​​a 2 k The order square matrix can be generated by the Sylvester algorithm, which uses the Kronecker Product to convert a 2 k-1 The basic matrix of order 2 is generated by continuous iteration k The specific formula of the Hadamard matrix is ​​as follows:

[0056] Basic matrix:

[0057] The formula for iterating the basic matrix is:

[0058] Among them, 2 k represents the order of the matrix, Represents the Kronecker product, where H2 is a basic matrix consisting of +1 and -1 elements, generated by the iterative formula Order square array. It also has the properties of the H1 matrix. For example, H4 can be obtained by the iterative formula:

[0059] Iterate 1 time:

[0060] In actual use, the Each row vector of Each column of is used as a Hadamard cluster center (i.e., the preset class center required for deep hash model training).

[0061] The multiple Hadamard cluster centers generated based on the Hadamard matrix are mutually orthogonal and bit-balanced, which can maximize the information entropy between multiple Hadamard cluster centers (hash codes with the same length as the hash code output by the deep hash model). Multiple Hadamard cluster centers are used as preset class centers to train the deep hash model to improve the accuracy of image retrieval.

[0062] Furthermore, in one embodiment, obtaining the multiple preset class centers based on the multiple Hadamard cluster centers may include: if the number of the multiple Hadamard cluster centers is less than the number of data categories in the training data set, cluster centers are generated using Bernoulli distribution to obtain multiple mutually orthogonal and bit-balanced Bernoulli cluster centers, wherein a generation probability and constraint conditions are set for the Bernoulli distribution, the generation probability is one-half, and the constraint condition is the distance constraint value between the class center generated by the Bernoulli distribution and the Hadamard cluster center; obtaining the multiple preset class centers based on the multiple Hadamard cluster centers and the multiple Bernoulli cluster centers.

[0063] When the number of data categories in the training data set is greater than the number of multiple Hadamard cluster centers that can be generated based on the Hadamard matrix (for example, the training data set includes data categories from Class1 to Class7, and the number of Hadamard cluster centers generated is 4), only more Bernoulli cluster centers can be generated as preset class centers using the Bernoulli distribution.

[0064] The cluster centers after random sampling of the Bernoulli distribution do not overlap with each other. During the incremental learning process of the deep hashing model, Bernoulli cluster centers can be added as preset cluster centers for newly added category data at any time.

[0065] The Bernoulli distribution is a discrete probability distribution, i.e., a 0 / 1 distribution. Therefore, the Bernoulli distribution can easily generate hash codes with only two code values: -1 and +1. Assuming that the probability of generating a code value of +1 is p, and the probability of generating a code value of -1 is q = 1-p, we can obtain: P(x = 1) = p, P(x = -1) = 1-p, 0 ≤ p ≤ 1; the quality function can be expressed by the piecewise function f(x) as: If the random variable x follows a Bernoulli distribution with parameter p, then the expectation E can be obtained as:

[0066] In order to make the Bernoulli cluster centers generated by the Bernoulli distribution (the hash code with the same length as the hash code output by the deep hash model) have the excellent property of bit balance, so as to increase the information entropy that each Bernoulli cluster center can carry, the probability of generation of the Bernoulli distribution is set to p = 0.5 (that is, the generation probability is set to one-half), that is, the expectation E(x) = 0, that is, a bit-balanced hash code can be generated as the Bernoulli cluster center.

[0067] At the same time, in order to prevent the randomly generated Bernoulli cluster centers from being too close to the generated Hadamard cluster centers, we can further set constraints for the Bernoulli distribution: i∈[1,j]; where j is the number of generated Hadamard cluster centers, c i represents the i-th generated Hadamard cluster center, @ represents the vector inner product, c Bernoulli represents the newly generated Bernoulli cluster center, and k represents the length of the hash code. Substituting this constraint into the Hamming distance formula yields:

[0068] Specifically, this constraint, converted to the Hamming distance formula, means that the distance between the newly generated cluster centers and the previously generated cluster centers in the Bernoulli distribution must be at least k / 4, ensuring that the newly generated cluster centers have an appropriate Hamming distance. Furthermore, because the k / 4 constraint is half the distance between the centers in the Hadamard matrix, the number of pre-set cluster centers that can be generated by the Bernoulli distribution is also guaranteed.

[0069] Furthermore, the multiple Hadamard cluster centers and the multiple Bernoulli cluster centers obtained can be used as the multiple preset class centers. It is understood that in other embodiments, if the number of data categories in the training data set is equal to the number of Hadamard cluster centers that can be generated based on the Hadamard matrix, the multiple Hadamard cluster centers obtained can be directly used as the multiple preset class centers.

[0070] In one embodiment, performing end-to-end incremental learning training on the deep hash model to be trained based on the query data and the plurality of preset class centers until a predetermined training end condition is met to obtain the trained preset deep hash model includes:

[0071] The sample image in the query data is input into the deep hash model to be trained to obtain the sample hash code of the sample image output by the deep hash model to be trained; if the sample image comes from the incremental data set, a first loss is calculated based on the sample hash code and the multiple preset class centers; if the sample image comes from the original data set, a second loss is calculated based on the sample hash code and the multiple preset class centers; according to the first loss or the second loss, the parameters of the deep hash model to be trained are adjusted until the predetermined training end condition is met, thereby obtaining the preset deep hash model after training.

[0072] During incremental learning, the original dataset used to train the deep hashing model to be trained is used along with the newly added incremental dataset. During incremental learning, the model not only needs to learn the incremental data information to ensure that it can properly retrieve the incremental data, but also needs to maintain its ability to remember the original data distribution to avoid degradation in original data retrieval performance after incremental learning.

[0073] Therefore, query data is randomly sampled from a mixture of the original dataset and the incremental dataset. Different processing is performed on the sample images in the original dataset and the incremental dataset: if the sample image comes from the incremental dataset, a first loss is calculated based on the sample hash code and the multiple preset class centers; if the sample image comes from the original dataset, a second loss is calculated based on the sample hash code and the multiple preset class centers. Then, based on the first loss or the second loss, the parameters of the deep hash model to be trained are adjusted until the predetermined training termination conditions are met, completing incremental learning and obtaining the trained preset deep hash model.

[0074] Furthermore, in one embodiment, the first loss is calculated based on the sample hash code and multiple preset class centers, and specifically may include: obtaining a center quantization loss based on the sample hash code and the preset class center corresponding to the data category of the sample image; obtaining a similarity loss based on the sample hash code and the image feature data of the incremental image in the incremental data set; and obtaining the first loss by fusing the center quantization loss and the similarity loss.

[0075] When the input sample images are incremental data (i.e., the sample images come from an incremental dataset), the model is trained using the same process as the original data. By learning similarity information from the incremental data, the model clusters the newly added categories of incremental data toward the newly preset cluster centers, ensuring that the cluster centers of the original and incremental data do not overlap during incremental learning.

[0076] The specific loss function can be: Among them, α is a hyperparameter, is the incremental hash database B inc The label hash code of the i-th incremental image in the incremental hash database B inc A set of preset label hash codes for each image in the incremental dataset; is the sample hash code of the jth sample image in the query data Q; ω is the incremental dataset, ζ∈ω is the set of sample images in the query data collected from the incremental dataset; τ is the set of preset class centers, c j is the jth preset class center; when Belongs to the preset center cj Time Tags Otherwise 0; center quantization loss s ij Represents the similarity label between the jth sample image of the query data and the ith original image of the original dataset. When s ij =1 means similarity, s ij = 0 indicates dissimilarity; k is the length of the hash code; similarity loss

[0077] Based on the above loss function, the first loss can be calculated, where L similar is the similarity loss, L center Quantify the loss for the center. center It can guide sample images of different data categories to cluster to their corresponding preset class centers to ensure that the original data and incremental data class centers do not overlap during incremental learning of the model. At the same time, Loss center It can also reduce the quantitative error of the model. similar The asymmetric hashing method can be used to associate the relationship between each sample image and the incremental images in the incremental dataset, thereby improving the model's ability to cluster data of different categories.

[0078] Furthermore, in one embodiment, the second loss calculated based on the sample hash code and the multiple preset class centers may specifically include: calculating a center quantization loss based on the sample hash code and the preset class center corresponding to the data category of the sample image; calculating a similarity loss based on the sample hash code and the image feature data of the incremental image in the incremental data set; calculating a memory loss based on the sample hash code and the image feature data of the original image in the original data set; and fusing the center quantization loss, the similarity loss and the memory loss to obtain the second loss.

[0079] In incremental learning, when the input sample image is an image in the original dataset, in order to ensure that the model maintains the model's ability to remember the original data distribution when learning incremental data, and to resist the phenomenon that the concept drift causes the model's retrieval performance of the original data to deteriorate, the memory loss Loss is further calculated. keep , this loss can improve the model's ability to remember the data distribution in the original dataset during incremental learning.

[0080] Specifically, we can use the similarity loss L similar The loss function structure is used as the memory loss L keep The structure of the loss function, specifically, the memory loss L keep The structure of the loss function can be:

[0081] in, is the original hash database B o The label hash code of the i-th original image in the original hash database B o is the set of label hash codes of the original images in the original dataset; is the sample hash code of the jth sample image in the query data Q. φ is the original dataset, and δ∈φ is the set of sample images in the query data sampled from the original dataset.

[0082] Through memory loss L keep The loss function can calculate the dot product of the image hash code of the sample image in the query data and the label hash code of the original image in the original dataset before the incremental learning of the model, that is, minimize the hash code error between different or same category original data, thereby constraining the model to maintain the retrieval results of the original data after incremental learning.

[0083] When learning incremental data through integration, when the input sample image is the image in the original dataset, the loss function is:

[0084] When the model learns incremental data, it will use L keep Maintain the model's ability to remember the original data distribution, while using L center The new category data in the incremental data are clustered to their corresponding preset class centers, and L similar The model is used to learn the relationship between the current batch of incremental data and other batches of incremental data, thereby completing the incremental learning function of the model.

[0085] To facilitate better implementation of the image retrieval method provided in the embodiments of this application, the embodiments of this application also provide an image retrieval device based on the aforementioned image retrieval method. The meanings of the terms herein are the same as those in the aforementioned image retrieval method. For specific implementation details, please refer to the description in the method embodiments. Figure 4 shows a block diagram of an image retrieval device according to one embodiment of the present application.

[0086] As shown in Figure 4, the image retrieval device 400 may include: an acquisition module 410 can be used to acquire the image to be retrieved; a processing module 420 can be used to process the image to be retrieved using a preset deep hash model to obtain an image hash code corresponding to the image to be retrieved, wherein the preset deep hash model is obtained by end-to-end incremental learning training based on multiple preset class centers and query data, and the multiple preset class centers are pre-constructed before model training. During the incremental learning training process, the original image or incremental image in the query data is clustered to the preset class centers corresponding to its data category; the matching module 430 can be used to match the image hash code with a preset hash code in a hash code set to obtain a preset hash code matched by the image hash code; the output module 440 can be used to obtain a matching image of the image to be retrieved based on a preset image corresponding to the preset hash code matched by the image hash code.

[0087] In some embodiments of the present application, the device also includes a training module, which is used to: pre-construct class centers before training the preset deep hash model to obtain multiple preset class centers, wherein each preset class center corresponds to a data category; sample training data from the original data set and the incremental data set to obtain query data; perform end-to-end incremental learning training on the deep hash model to be trained based on the query data and the multiple preset class centers until the predetermined training end conditions are met, thereby obtaining the trained preset deep hash model, wherein, during the incremental learning training process, the original data or incremental data in the query data are clustered to the preset class centers corresponding to their data categories.

[0088] In some embodiments of the present application, the training module is used to: generate cluster centers using a Hadamard matrix to obtain multiple Hadamard cluster centers that are mutually orthogonal and bit-balanced; and obtain the multiple preset cluster centers based on the multiple Hadamard cluster centers.

[0089] In some embodiments of the present application, the training module is used to: if the number of the multiple Hadamard cluster centers is less than the number of data categories in the training data set, generate cluster centers using the Bernoulli distribution method to obtain multiple mutually orthogonal and bit-balanced Bernoulli cluster centers, wherein the generation probability and constraint conditions are set for the Bernoulli distribution, the generation probability is one-half, and the constraint condition is the distance constraint value between the class center generated by the Bernoulli distribution and the Hadamard cluster center; based on the multiple Hadamard cluster centers and the multiple Bernoulli cluster centers, the multiple preset class centers are obtained.

[0090] In some embodiments of the present application, the training module is used to: input the sample image in the query data into the deep hash model to be trained, and obtain the sample hash code of the sample image output by the deep hash model to be trained; if the sample image comes from the incremental data set, a first loss is calculated based on the sample hash code and the multiple preset class centers; if the sample image comes from the original data set, a second loss is calculated based on the sample hash code and the multiple preset class centers; according to the first loss or the second loss, the parameters of the deep hash model to be trained are adjusted until the predetermined training end conditions are met, thereby obtaining the preset deep hash model after training.

[0091] In some embodiments of the present application, the training module is used to: calculate a central quantization loss based on the preset class center corresponding to the sample hash code and the data category of the sample image; calculate a similarity loss based on the sample hash code and the image feature data of the incremental image in the incremental data set; and fuse the central quantization loss and the similarity loss to obtain the first loss.

[0092] In some embodiments of the present application, the training module is used to: calculate a central quantization loss based on the sample hash code and the preset class center corresponding to the data category of the sample image; calculate a similarity loss based on the sample hash code and the image feature data of the incremental image in the incremental data set; calculate a memory loss based on the sample hash code and the image feature data of the original image in the original data set; and fuse the central quantization loss, the similarity loss and the memory loss to obtain the second loss.

[0093] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0094] In addition, an embodiment of the present application further provides an electronic device, as shown in FIG5 . FIG5 shows a block diagram of an electronic device according to an embodiment of the present application. Specifically:

[0095] The electronic device may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, and an input unit 504. Those skilled in the art will appreciate that the electronic device structure shown in FIG5 does not limit the electronic device and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0096] Processor 501 is the control center of the electronic device. It utilizes various interfaces and circuits to connect the various components of the entire computer device. By running or executing software programs and / or modules stored in memory 502 and accessing data stored in memory 502, it performs various computer device functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, processor 501 may include one or more processing cores; preferably, processor 501 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interfaces, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 501.

[0097] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0098] The electronic device also includes a power supply 503 for supplying power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 503 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0099] The electronic device may further include an input unit 504, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0100] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 501 in the electronic device will load the executable files corresponding to one or more computer program processes into the memory 502 according to the following instructions, and the processor 501 will run the computer program stored in the memory 502, thereby realizing the various functions of the aforementioned embodiments of the present application. For example, the processor 501 may perform the following steps:

[0101] Acquire an image to be retrieved; use a preset deep hash model to process the image to be retrieved to obtain an image hash code corresponding to the image to be retrieved, wherein the preset deep hash model is obtained by end-to-end incremental learning training based on multiple preset class centers and query data, and the multiple preset class centers are pre-constructed before model training. During the incremental learning training, the original image or incremental image in the query data is clustered to the preset class centers corresponding to its data category; match the image hash code with a preset hash code in a hash code set to obtain a preset hash code that matches the image hash code; obtain a matching image of the image to be retrieved according to a preset image corresponding to the preset hash code that matches the image hash code.

[0102] In some embodiments of the present application, it also includes: pre-constructing class centers before training the preset deep hash model to obtain multiple preset class centers, wherein each preset class center corresponds to a data category; sampling training data from the original data set and the incremental data set to obtain query data; performing end-to-end incremental learning training on the deep hash model to be trained based on the query data and the multiple preset class centers until the predetermined training end conditions are met, thereby obtaining the trained preset deep hash model, wherein, during the incremental learning training process, the original data or incremental data in the query data are clustered to the preset class centers corresponding to their data categories.

[0103] In some embodiments of the present application, the class center is pre-constructed before the preset deep hash model is trained to obtain multiple preset class centers, including: using a Hadamard matrix to generate cluster centers to obtain multiple Hadamard cluster centers that are mutually orthogonal and bit-balanced; and obtaining the multiple preset class centers based on the multiple Hadamard cluster centers.

[0104] In some embodiments of the present application, the multiple preset class centers are obtained based on the multiple Hadamard cluster centers, including: if the number of the multiple Hadamard cluster centers is less than the number of data categories in the training data set, cluster centers are generated using Bernoulli distribution to obtain multiple mutually orthogonal and bit-balanced Bernoulli cluster centers, wherein a generation probability and constraint conditions are set for the Bernoulli distribution, the generation probability is one-half, and the constraint condition is the distance constraint value between the class center generated by the Bernoulli distribution and the Hadamard cluster center; the multiple preset class centers are obtained based on the multiple Hadamard cluster centers and the multiple Bernoulli cluster centers.

[0105] In some embodiments of the present application, the end-to-end incremental learning training of the deep hash model to be trained based on the query data and the multiple preset class centers is performed until a predetermined training end condition is met to obtain the trained preset deep hash model, including: inputting the sample image in the query data into the deep hash model to be trained, and obtaining a sample hash code of the sample image output by the deep hash model to be trained; if the sample image comes from the incremental data set, a first loss is calculated based on the sample hash code and the multiple preset class centers; if the sample image comes from the original data set, a second loss is calculated based on the sample hash code and the multiple preset class centers; according to the first loss or the second loss, the parameters of the deep hash model to be trained are adjusted until the predetermined training end condition is met to obtain the trained preset deep hash model.

[0106] In some embodiments of the present application, the first loss is calculated based on the sample hash code and the multiple preset class centers, including: calculating a center quantization loss based on the sample hash code and the preset class center corresponding to the data category of the sample image; calculating a similarity loss based on the sample hash code and the image feature data of the incremental image in the incremental data set; and fusing the center quantization loss and the similarity loss to obtain the first loss.

[0107] In some embodiments of the present application, the second loss is calculated based on the sample hash code and the multiple preset class centers, including: calculating a center quantization loss based on the sample hash code and the preset class center corresponding to the data category of the sample image; calculating a similarity loss based on the sample hash code and the image feature data of the incremental image in the incremental data set; calculating a memory loss based on the sample hash code and the image feature data of the original image in the original data set; and fusing the center quantization loss, the similarity loss and the memory loss to obtain the second loss.

[0108] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by a computer program, or by controlling related hardware through a computer program. The computer program may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0109] To this end, an embodiment of the present application further provides a storage medium storing a computer program, which can be loaded by a processor to execute the steps of any method provided in the embodiment of the present application.

[0110] The storage medium may be a computer-readable storage medium, and the storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0111] Since the computer program stored in the storage medium can execute the steps of any method provided in the embodiments of the present application, the beneficial effects that can be achieved by the method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0112] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0113] It should be understood that the present application is not limited to the embodiments that have been described above and shown in the accompanying drawings, but various modifications and changes may be made without departing from the scope thereof.

Claims

1. An image retrieval method, wherein: include: Get the image to be retrieved; The image to be retrieved is processed using a preset deep hash model to obtain an image hash code corresponding to the image to be retrieved, wherein the preset deep hash model is obtained by end-to-end incremental learning training based on multiple preset class centers and query data, the multiple preset class centers are pre-constructed before model training, and during the incremental learning training process, the original image or incremental image in the query data is clustered to the preset class centers corresponding to its data category; Matching the image hash code with a preset hash code in the hash code set to obtain a preset hash code that matches the image hash code; A matching image of the image to be retrieved is obtained according to a preset image corresponding to a preset hash code that matches the image hash code.

2. The method according to claim 1, wherein The preset deep hash model is trained in the following manner: Before training the preset deep hash model, a class center is pre-constructed to obtain a plurality of preset class centers, wherein each preset class center corresponds to a data category; Sampling training data from the original dataset and incremental dataset to obtain query data; Based on the query data and the multiple preset class centers, end-to-end incremental learning training is performed on the deep hash model to be trained until a predetermined training end condition is met, so as to obtain the trained preset deep hash model, wherein, during the incremental learning training process, the original data or incremental data in the query data are clustered to the preset class centers corresponding to their data categories.

3. The method according to claim 2, wherein: The class center is pre-built before the preset deep hash model is trained to obtain multiple preset class centers, including: The Hadamard matrix is ​​used to generate cluster centers, and multiple Hadamard cluster centers that are mutually orthogonal and balanced are obtained; The multiple preset cluster centers are obtained according to the multiple Hadamard cluster centers.

4. The method according to claim 3, wherein: The step of obtaining the plurality of preset cluster centers according to the plurality of Hadamard cluster centers includes: If the number of the multiple Hadamard cluster centers is less than the number of data categories in the training data set, cluster centers are generated using Bernoulli distribution to obtain multiple mutually orthogonal and bit-balanced Bernoulli cluster centers, wherein a generation probability and a constraint condition are set for the Bernoulli distribution, the generation probability is one-half, and the constraint condition is a distance constraint value between the class center generated by the Bernoulli distribution and the Hadamard cluster center; The plurality of preset cluster centers are obtained according to the plurality of Hadamard cluster centers and the plurality of Bernoulli cluster centers.

5. The method according to claim 2, wherein: The method of performing end-to-end incremental learning training on the deep hash model to be trained based on the query data and the plurality of preset class centers until a predetermined training end condition is met to obtain the trained preset deep hash model includes: Inputting the sample image in the query data into the deep hash model to be trained, and obtaining a sample hash code of the sample image output by the deep hash model to be trained; If the sample image is from the incremental dataset, calculating a first loss based on the sample hash code and the plurality of preset class centers; If the sample image is derived from the original dataset, a second loss is calculated based on the sample hash code and the plurality of preset class centers; According to the first loss or the second loss, the parameters of the deep hash model to be trained are adjusted until a predetermined training end condition is met, thereby obtaining the trained preset deep hash model.

6. The method according to claim 5, wherein: The calculating a first loss based on the sample hash code and the plurality of preset class centers includes: A center quantization loss is calculated based on the sample hash code and the preset class center corresponding to the data class of the sample image; Calculating a similarity loss based on the sample hash code and image feature data of the incremental image in the incremental dataset; The center quantization loss and the similarity loss are fused to obtain the first loss.

7. The method according to claim 5, wherein: The calculating the second loss based on the sample hash code and the plurality of preset class centers includes: A center quantization loss is calculated based on the sample hash code and the preset class center corresponding to the data class of the sample image; Calculating a similarity loss based on the sample hash code and image feature data of the incremental image in the incremental dataset; Calculating memory loss based on the sample hash code and image feature data of the original image in the original data set; The second loss is obtained by fusing the center quantization loss, the similarity loss, and the memory loss.

8. The method according to claim 1, wherein The process of processing the image to be retrieved by using a preset deep hash model to obtain an image hash code corresponding to the image to be retrieved includes: The convolution and pooling layer performs convolution and pooling processing on the image to be retrieved to obtain the first layer of features; The first layer features are fully connected by the fully connected layer to obtain the second layer features; The second layer features are quantized at the hash layer using a hash function to obtain an image hash code corresponding to the image to be retrieved.

9. The method according to claim 3, wherein: The step of obtaining the plurality of preset cluster centers according to the plurality of Hadamard cluster centers includes: If the number of data categories in the training data set is equal to the number of the multiple Hadamard cluster centers, the multiple Hadamard cluster centers are used as the multiple preset class centers.

10. The method according to claim 6, wherein: The fusing the center quantization loss and the similarity loss to obtain the first loss includes: According to the formula L1=L similar +αL center The first loss is obtained by fusing the center quantization loss and the similarity loss, where L1 refers to the first loss, L similar Refers to the similarity loss, L center Refers to the center quantization loss, and α is a hyperparameter.

11. The method according to claim 7, wherein: The fusing of the center quantization loss, the similarity loss, and the memory loss to obtain the second loss includes: According to the formula L2=αL center +L similar +L keep The second loss is obtained by fusing the center quantization loss, the similarity loss and the memory loss, where L similar Refers to the similarity loss, L center Refers to the central quantization loss, L keep refers to the memory loss, L2 refers to the second loss, and α is a hyperparameter.

12. An image retrieval device, wherein: include: An acquisition module, used to acquire the image to be retrieved; a processing module, configured to process the image to be retrieved using a preset deep hash model to obtain an image hash code corresponding to the image to be retrieved, wherein the preset deep hash model is obtained by performing end-to-end incremental learning training based on multiple preset class centers and query data, the multiple preset class centers being pre-constructed before model training, and during the incremental learning training process, the original image or incremental image in the query data is clustered towards the preset class centers corresponding to its data category; a matching module, configured to match the image hash code with a preset hash code in a hash code set to obtain a preset hash code that matches the image hash code; The output module is configured to obtain a matching image of the image to be retrieved based on a preset image corresponding to a preset hash code matched by the image hash code.

13. The device according to claim 12, wherein The device also includes a training module, which is used to: pre-construct class centers before training the preset deep hash model to obtain multiple preset class centers, wherein each preset class center corresponds to a data category; sample training data from the original data set and the incremental data set to obtain query data; perform end-to-end incremental learning training on the deep hash model to be trained based on the query data and the multiple preset class centers until a predetermined training end condition is met, thereby obtaining the trained preset deep hash model, wherein, during the incremental learning training process, the original data or incremental data in the query data are clustered to the preset class centers corresponding to their data categories.

14. The device according to claim 13, wherein The training module is used to: generate cluster centers using a Hadamard matrix to obtain a plurality of Hadamard cluster centers that are mutually orthogonal and bit-balanced; and obtain the plurality of preset cluster centers based on the plurality of Hadamard cluster centers.

15. The device according to claim 14, wherein The training module is used to: if the number of the multiple Hadamard cluster centers is less than the number of data categories in the training data set, generate cluster centers using the Bernoulli distribution method to obtain multiple mutually orthogonal and bit-balanced Bernoulli cluster centers, wherein the generation probability and constraint conditions are set for the Bernoulli distribution, the generation probability is one-half, and the constraint condition is the distance constraint value between the class center generated by the Bernoulli distribution and the Hadamard cluster center; based on the multiple Hadamard cluster centers and the multiple Bernoulli cluster centers, the multiple preset class centers are obtained.

16. The device according to claim 13, wherein The training module is configured to: input the sample image in the query data into the deep hash model to be trained, and obtain a sample hash code of the sample image output by the deep hash model to be trained; If the sample image is from the incremental dataset, calculating a first loss based on the sample hash code and the plurality of preset class centers; If the sample image comes from the original data set, a second loss is calculated based on the sample hash code and the multiple preset class centers; according to the first loss or the second loss, the parameters of the deep hash model to be trained are adjusted until the predetermined training end conditions are met, thereby obtaining the trained preset deep hash model.

17. The device according to claim 16, wherein The training module is used to: calculate a central quantization loss based on the sample hash code and the preset class center corresponding to the data category of the sample image; calculate a similarity loss based on the sample hash code and the image feature data of the incremental image in the incremental data set; and fuse the central quantization loss and the similarity loss to obtain the first loss.

18. The device according to claim 16, wherein The training module is used to: calculate a central quantization loss based on the sample hash code and the preset class center corresponding to the data category of the sample image; calculate a similarity loss based on the sample hash code and the image feature data of the incremental image in the incremental data set; calculate a memory loss based on the sample hash code and the image feature data of the original image in the original data set; and fuse the central quantization loss, the similarity loss and the memory loss to obtain the second loss.

19. A storage medium, wherein: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the method according to any one of claims 1 to 11.

20. An electronic device, wherein: include: a memory storing a computer program; A processor reads a computer program stored in a memory to execute the method according to any one of claims 1 to 11.

Images