Similar image retrieval method and system of deep hash model based on binary network

Abstract

The invention discloses a similar image retrieval method and system of a deep hash model based on a binary network, in the scheme provided by the invention, the deep hash model based on the binary network is constructed and is used for quickly retrieving image data, and compared with the existing scheme, the method and the system have the advantages that the retrieval efficiency is greatly improved; according to the scheme provided by the invention, the problem of calculation overhead in a model inference process is considered, and the storage size of the model and the calculation complexity are greatly reduced based on the binary network, so that the model can be deployed on resource-limited equipment, and meanwhile, due to the limited expression capability of the binary network, the calculation complexity is greatly reduced. The invention further provides a semantic-based Hash center self-adaptive module and a two-stage training method, so that the effectiveness of a retrieval result is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of similar image retrieval, and in particular to a similar image retrieval method and system based on a deep hash model of a binary network. Background Art

[0002] In recent years, with the vigorous development of online technology, image data has increased exponentially, and users have access to massive amounts of image data. Similar image retrieval systems (or content-based image retrieval systems) can find images similar to the query image in a large number of image resources based on the image query provided by the user, which greatly alleviates the degree of information overload and is therefore widely used in various applications. And with the rapid development of technology, similar image retrieval systems have undergone improvements from manual feature methods to deep learning methods, which has continuously improved the accuracy of retrieval. However, the efficiency of retrieval is still an inevitable key issue in similar image retrieval systems, which is closely related to the user experience and the burden of the system. Therefore, how to improve the efficiency of retrieval is a research problem that needs to be urgently solved in similar image retrieval systems.

[0003] Around this research problem, researchers have proposed a variety of methods, among which the deep hashing method is a solution that has received much attention in recent years. Its main idea is to use deep network models such as ResNet, VGG, etc. and some binarization operations to map image data into binary representations, which are also called hash codes. The images will be stored in the hash buckets corresponding to the hash codes. On the one hand, the similarity of two hash codes can be quickly calculated by using computer instructions XOR and POPCOUNT. On the other hand, semantic hash indexes can also be used to quickly return related images from a set of candidate images. Therefore, semantic hashing technology can greatly alleviate the problem of search efficiency.

[0004] However, current semantic hashing methods only rely on the characteristics of hash codes to speed up the search process, but ignore the computational efficiency of the model during inference. Specifically, in order to ensure the accuracy of the search, existing semantic hashing work usually uses some large model skeletons to extract image features, which increases the computational overhead of the deep hash model in the model part, making it difficult to deploy in resource-constrained devices. Therefore, in the design of deep hash models, how to reduce the computational complexity of the deep hash model inference process while ensuring the retrieval quality is a very important issue. Summary of the invention

[0005] The purpose of the present invention is to provide a similar image retrieval method and system based on a deep hash model of a binary network, which can reduce the amount of calculation in the deep hash model inference process while ensuring the retrieval quality, so that it can be applied to resource-constrained devices.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] A similar image retrieval method based on a deep hash model of a binary network, comprising:

[0008] Construct a deep hashing model based on binary network;

[0009] A semantic-based hash center adaptive module is introduced to train the binary network-based deep hash model. A classification model is provided in the semantic-based hash center adaptive module. The classification model is used to obtain the semantic information of the category and generate the hash center. The training process includes two stages. In the first stage, the classification loss of the classification model is used to train the semantic-based hash center adaptive module. The second stage includes a space exploration stage and a code aggregation stage. In the space exploration stage, the hash center alignment loss is calculated using the hash center and the class binary hash representation output by the binary network-based deep hash model, and the classification loss is combined to jointly train the binary network-based deep hash model and the semantic-based hash center adaptive module. In the code aggregation stage, the binary network-based deep hash model is used to obtain a class binary hash representation set, and the centroid vector of each semantic category is calculated, and the extrapolation method is used for processing. The extrapolation aggregation loss is calculated using the processing result, and the classification loss and the hash center alignment loss are combined to jointly train the binary network-based deep hash model and the semantic-based hash center adaptive module.

[0010] A binary hash representation set of image data in a database is obtained using a trained binary network-based deep hash model, and a semantic hash index is constructed; and a hash code of a query image is obtained using a trained binary network-based deep hash model, and image retrieval is performed in combination with the semantic hash index.

[0011] A similar image retrieval system based on a deep hashing model of a binary network, comprising:

[0012] A model building unit, used to build a deep hashing model based on a binary network;

[0013] A model training unit is used to introduce a semantic-based hash center adaptive module to train the binary network-based deep hash model, wherein the semantic-based hash center adaptive module is provided with a classification model, and the semantic information of the category is obtained through the classification model and a hash center is generated; the training process includes two stages, in the first stage, the semantic-based hash center adaptive module is trained using the classification loss of the classification model; the second stage includes a space exploration stage and a code aggregation stage, in the space exploration stage, the hash center alignment loss is calculated using the hash center and the class binary hash representation output by the binary network-based deep hash model, and the classification loss is combined to jointly train the binary network-based deep hash model and the semantic-based hash center adaptive module; in the code aggregation stage, the class binary hash representation set is obtained using the binary network-based deep hash model, the centroid vector of each semantic category is calculated, and the extrapolation method is used for processing, and the extrapolation aggregation loss is calculated using the processing result, and the classification loss and the hash center alignment loss are combined to jointly train the binary network-based deep hash model and the semantic-based hash center adaptive module;

[0014] An image retrieval unit is used to obtain a binary hash representation set of image data in a database using a trained deep hash model based on a binary network, and to construct a semantic hash index; and to obtain a hash code of a query image using a trained deep hash model based on a binary network, and to perform image retrieval in combination with the semantic hash index.

[0015] A processing device, comprising: one or more processors; a memory for storing one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0017] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

[0018] It can be seen from the technical solution provided by the present invention that a deep hash model based on a binary network is constructed and used to quickly retrieve image data. Compared with the existing solutions, the present invention takes into account the computational overhead problem in the model inference process. Based on the binary network, the storage size and computational complexity of the model are greatly reduced, so that it can be deployed on resource-constrained devices. At the same time, because the expression ability of the binary network is limited, the present invention proposes a semantic-based hash center adaptive module and a two-stage training method to ensure the effectiveness of the retrieval results. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0020] Figure 1 A flowchart of a similar image retrieval method based on a deep hash model of a binary network provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of a similar image retrieval method based on a deep hash model of a binary network provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of a training scheme for a deep hash model based on a binary network provided in an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of a similar image retrieval system based on a deep hash model of a binary network provided by an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0026] First, the terms that may be used in this article are explained as follows:

[0027] The term “and / or” means that either or both of them can be realized at the same time. For example, X and / or Y means both “X” or “Y” and “X and Y”.

[0028] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0029] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0030] The following is a detailed description of a similar image retrieval method and system based on a deep hash model of a binary network provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field. If no specific conditions are specified in the embodiments of the present invention, they shall be carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer.

[0031] Embodiment 1

[0032] The embodiment of the present invention provides a similar image retrieval method based on a deep hash model of a binary network, such as Figure 1 As shown, it mainly includes the following steps:

[0033] Step 1: Build a deep hashing model based on binary network.

[0034] In the embodiment of the present invention, the deep hash model based on the binary network includes: a binary network image feature extractor and a hash layer; wherein the image feature v is extracted by the binary network image feature extractor, and a binary-like hash representation is obtained by the hash layer. During the training phase, the class binary hash representation is used for training. After the training is completed, the class binary hash representation is converted into a binary hash code h through a symbolic function.

[0035] Step 2: Introduce a semantic-based hash center adaptive module to train the binary network-based deep hash model.

[0036] In an embodiment of the present invention, a classification model is provided in the semantic-based hash center adaptive module, and the semantic information of the category is obtained and a hash center is generated through the classification model; the training process includes two stages, in the first stage, the classification loss of the classification model is used to train the semantic-based hash center adaptive module; the second stage includes a space exploration stage and a code aggregation stage, in the space exploration stage, the hash center alignment loss is calculated using the hash center and the class binary hash representation output by the deep hash model based on the binary network, and the deep hash model based on the binary network and the semantic-based hash center adaptive module are jointly trained in combination with the classification loss; in the code aggregation stage, the deep hash model based on the binary network is used to obtain a class binary hash representation set, the centroid vector of each semantic category is calculated, and the extrapolation method is used for processing, and the extrapolation aggregation loss is calculated using the processing result, and the classification loss and the hash center alignment loss are combined to jointly train the deep hash model based on the binary network and the semantic-based hash center adaptive module.

[0037] Step 3: Perform image retrieval through the trained binary network-based deep hashing model.

[0038] In an embodiment of the present invention, a binary hash representation set of image data in a database is obtained using a trained binary network-based deep hash model, and a semantic hash index is constructed; and a hash code of a query image is obtained using a trained binary network-based deep hash model, and image retrieval is performed in combination with the semantic hash index.

[0039] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.

[0040] 1. Principle overview.

[0041] The embodiment of the present invention provides a similar image retrieval solution based on a deep hash model of a binary network, which overcomes the problem that the existing deep model has a large amount of calculation during deployment and cannot be applied to resource-constrained devices. The present invention provides a deep hash model based on a binary network, which is mainly used to generate a hash representation (quasi-binary hash representation) with semantic information from image data, and reduces the computational overhead in the generation process, thereby expanding the application scenarios of semantic hashing, such as application in some resource-constrained devices. Based on the current deep hash model, the present invention can use any binary network to replace the original image representation extraction module. However, due to the weak representation ability of the binary network, the accuracy of the generated hash code is greatly reduced compared to the original real-valued network. Therefore, according to the characteristics of the binary network, the case of the present invention designs a semantic-based hash center adaptive module and a two-stage training method to ensure the final retrieval effect.

[0042] During training, image data and their corresponding semantic labels are obtained from the Internet, invalid data and data with high noise levels are removed, and the data is divided into a training set and a validation set.

[0043] In the first stage, the semantic-based hash center module is first trained, and cross entropy can be used as the target loss. The semantic center module in the case of the present invention can use any image feature extraction network, such as ResNet series, VGG series, or ViT series network. In the case of the present invention, ResNet50 is used as the image feature extraction network. Through the trained semantic-based hash center module, a hash center can be generated for each semantic.

[0044] In the second stage, a deep hash model based on a binary network is trained. The present invention generates a hash center from a semantic-based hash center module as the target of the corresponding semantic image, and adopts a two-stage approach to train to meet the special properties of the binary network in deep hashing. In the space exploration stage, the hash center loss is used to force the hash code to approach the hash center. In the code aggregation stage, an additional extrapolated aggregation loss target is added to promote the aggregation of hash codes of the same category. The binary network in the embodiment of the present invention can adopt any current binary network. ReActNet is used in the embodiment of the present invention, and other binary networks are also applicable.

[0045] like Figure 2 As shown, an example of the overall process of the solution of the present invention is shown: in the offline stage, through the training method proposed in the case of the present invention, after the training is completed, the image data in the database is passed through the trained binary network-based hash code generation model to obtain the corresponding hash code set, and the hash code set is used to build a semantic hash index. In the online stage, the query image is passed through the binary network-based hash code generation model to obtain a hash code, and the semantic hash index is used to quickly find related images and return the results.

[0046] 2. Detailed introduction of the plan.

[0047] 1. Problem definition and formalization.

[0048] In the embodiment of the present invention, two definitions need to be formalized, namely, a semantic hashing task based on a binary network and a semantic hashing retrieval framework.

[0049] Semantic hashing task based on binary network: define the image set as X = {x1, x2, ... x N}, where x j represents the jth image, j = 1, 2, ..., N, N represents the total number of images. The goal of semantic hashing based on binary networks is to learn a hash function f(x): x→h, which maps image x to a binary representation h∈{-1,1}b , where the symbol b represents the length of the binary representation. This binary representation h is also called a hash code. The hash function is implemented by a binary network.

[0050] Semantic hash retrieval framework: In the semantic hash retrieval framework, the goal is to find a set of images S(q) similar to the query image q given by the user. The framework consists of two stages, the offline stage and the online stage. In the offline stage, the image set X is mapped to a hash code using a binary network-based hash model f(x):x→h,x∈X, and a hash index is constructed. In the online stage, the same binary network-based hash model f(x):x→h is used. q Get the query hash code and quickly find related images through hash index.

[0051] 2. Data collection and preprocessing.

[0052] The examples in this invention use images as input datasets. Such data samples include CIFAR100, IMAGENET, and MSCOCO. In addition, various image datasets can also be collected through web crawling or offline as input data.

[0053] 3. Model establishment and training.

[0054] like Figure 3 As shown in the figure, the structure of the binary network-based deep hash model and the semantic-based hash center adaptation module, as well as the related training schemes, are demonstrated. The semantic-based hash center adaptation module uses the category semantics of the image to generate the hash center and makes adaptive adjustments during the training process. The first stage of training is mainly for the semantic-based hash center adaptation module. The second stage is divided into a spatial exploration training stage and a hash code aggregation training stage. The spatial exploration training stage uses the hash center loss target to force the hash code to approach the corresponding hash center. The hash code aggregation training stage adds an extrapolation aggregation loss target, which aims to aggregate the hash codes in the same category more closely in the hash space while keeping the hash codes of different categories at a sufficient distance.

[0055] (1) Deep hashing model based on binary network.

[0056] like Figure 2 As shown in part (a) of the figure, the deep model based on binary network consists of a binary network image feature extractor and a hash layer.

[0057] For example, a ReactNet model H(x):x→v can be used as a feature extractor, where v represents the features of the image. The feature v is then passed through a hash layer to obtain a hash-like representation

[0058] Exemplarily, the hashing layer may consist of a single-layer feed-forward network and a tanh(·) activation function.

[0059] The above process can be formalized as:

[0060]

[0061] Among them, W h and b h represents the learnable parameters in the hashing layer.

[0062] During the training phase of the model, the binary hash representation is used Participate in training to ensure that the gradient can be propagated in a certain direction. In the inference phase after the model is deployed, Through a symbol function sign, to obtain the final binary hash code

[0063] It is worth noting that if the traditional deep hashing training method is used, the accuracy of the hash code will be greatly reduced. This is because the representation ability of the binary network is greatly reduced compared to the real-valued network. Therefore, the case of the present invention introduces a special module and training method, which will be introduced in detail later.

[0064] (2) Semantic-based hash center adaptive module and its training scheme.

[0065] The main goal of the semantic-based hash center adaptation module is to i Category information y i ∈{0,1} m , get a corresponding hash center c′ i ={-1,1} b , thereby better guiding the distribution of generated hash codes in space, where m represents the total number of categories in the data. A direct approach is to convert the label y i Input a neural network, such as a multilayer perceptron (MLP), to obtain the hash center c′ i However, this method ignores the semantic information in the category. Therefore, the embodiment of the present invention introduces a semantic-based hash center adaptive module to make the generated hash center more accurate by adding the semantic information of the category.

[0066] First, a matrix of category semantics is obtained through a pre-training method Among them, R is the symbol of the real number set, C m Represents the category semantic vector w l,l=1,2,…,m. In the embodiment of the present invention, the category semantic matrix is ​​obtained through a classification task. The semantic-based hash center adaptive module is provided with a classification model, such as Figure 2 In the example shown in part (b) of , a ResNet50 network can be used as the image classification model. The last layer is the prediction layer of the category probability, and the second-to-last layer is the semantic layer of the category. Therefore, the network parameters of the second-to-last layer can be regarded as the semantic layer W of the category. Assume that the training batch data is Where |B| represents the size of the training batch. Set a classification task to train the network so that the category semantic layer can obtain the semantics of the category:

[0067]

[0068] in, Represents the predicted probability of the classification model on the lth category, which is the prediction result output by the classification model The lth element of il is the category information y i The lth element of i The real information belonging to the lth category. i It is represented in the form of bag-of-word (bag-of-word model).

[0069] After training the classification model, the network parameters of the penultimate layer can be used as the matrix of category semantics w l is the semantic vector of the lth category. For the i-th image x in the training batch B i , its category information y i After performing matrix multiplication with the matrix W of category semantics, the corresponding category feature w′ is obtained i :

[0070] w′ i =y i W

[0071] The category feature w′ is transformed into i Mapped to a probability vector p of the same dimension as the class binary hash representation i , the probability vector p i As the parameter of the multidimensional Bernoulli distribution, sampling is performed from it to obtain the hash center c′ i ={-1,1} b , where b represents the dimension of the binary hash representation of the class. Since the sampling process is not differentiable, a reparameterization method is used, and the complete representation of the relevant process is:

[0072] c′ i=sign(σ(F(w′ i ))-μ)

[0073] Wherein, F is a feedforward neural network, for example, it can be implemented using MLP. σ represents the sigmoid activation function. Sign represents the sign function, which maps values ​​greater than or equal to 0 to 1 and values ​​less than 0 to -1, and μ represents a value obtained from the uniform distribution U(0,1). In this representation, the derivative of sign is 0, so the derivative cannot be passed to the model. In an embodiment of the present invention, a straight-forward estimator (STE) is used to approximate the gradient. This allows the module to be trained in an end-to-end manner.

[0074] (3) Training plan for the second phase.

[0075] In the embodiment of the present invention, the second stage is divided into a space exploration stage and a code aggregation stage. These two stages are derived from an experimental observation during the research process of the present invention. When a binary network is used as an image feature extraction model for deep hashing, the hash codes corresponding to images of the same category will be dispersed first and then aggregated during the training process. Based on this feature, the present invention divides the training process of the deep hashing model based on the binary network into a space exploration training stage and a hash code aggregation training stage, and designs different training objectives for different stages.

[0076] (3.1) Space exploration training phase.

[0077] In the space exploration training phase, in order to meet the training characteristics of the binary network, only the image x i Generated hash code To get closer to the corresponding hash center c′ i In each training batch, we first pass the image x i After inputting the deep hash model based on binary network, the corresponding binary-like hash representation is obtained Then the category semantic matrix W is taken as input, and the hash center representation C = {c1, c2…, c m}, the specific process is as follows:

[0078] [c1,c2,…,c m ] T =sign(σ(F(W))-0.5)

[0079] Where T is the transposed symbol, and the sigmoid function σ represents the use of each row of the matrix. Then the following hash center alignment loss target is used:

[0080]

[0081] Among them, L hc is the hash center alignment loss, |B| is the number of images in the training batch B, φ is the similarity calculation function (e.g., cosine similarity), is the i-th image x in the training batch i Binary hash representation of c′ i is the i-th image x i The hash center of , τ represents the temperature hyperparameter, c l ∈C represents the hash center representation corresponding to l categories.

[0082] (3.2) Code aggregation stage.

[0083] In the code aggregation stage, in addition to using the hash center alignment loss target L mentioned above hc In addition to encouraging hash codes to be close to their hash centers, an extrapolated aggregation loss objective is introduced to enhance the aggregation between hash codes corresponding to images of the same semantic category. A basic idea is to The hash code distance of images of the same category is reduced in |B|. Where |B| represents the size of the training batch. Another thing worth considering is to ensure that samples of the same category in the batch are difficult samples to each other, thereby further promoting the aggregation of hash codes. To this end, the present invention proposes an extrapolated aggregation loss target.

[0084] Specifically: for data Several sets of enhanced view sets are obtained by data processing, and all enhanced view sets are used as training set B. new , the training set B is obtained through the deep hashing model based on the binary network new The binary hash representation set H new ,definition Indicates H new The class binary hash representation set belonging to category l in the class binary hash representation set H is calculated new The centroid vector of each semantic category in m represents the total number of categories.

[0085] like Figure 3 As shown in part (c), data augmentation is used to create each image x i =B generates two views and In this example, B new = B1∪B2. Of course, in practical applications, this can be expanded to more views, which is not limited in the present invention.

[0086] Among them, the centroid vector s corresponding to the lth categoryl The calculation method is:

[0087]

[0088] in, express The number of elements in , for An element in , i.e. a class binary hash representation;

[0089] Using extrapolation method, we can obtain The new representation h′ k , that is, Hard Positive Sample Representation, expressed as:

[0090]

[0091] Among them, Φ represents the extrapolation method, s l for The centroid vector of the corresponding semantic category, α~U(0,1)+1, that is, α is the value sampled from U(0,1)+1, U(0,1) represents a uniform distribution with an expected value of 0 and a variance of 1. Adding 1 at the end ensures that the range of α is between [1,2]. The above settings can ensure Thus increasing The variance of the distances between binary hash representations of the classes.

[0092] Finally, the extrapolated aggregation loss is calculated using the processing results, which is expressed as:

[0093]

[0094] Among them, L asd To extrapolate the aggregation loss, for Another element in D h is the distance calculation function.

[0095] The goal of the above extrapolated aggregation loss is not only to effectively utilize the information of other images sharing the same category in the batch, but also to increase the variance of the hash code, thereby improving the effect of hash code aggregation. In addition, when the batch size |B| is less than the number of categories m, some categories may have only one image or a small number of images in each training batch. In order to solve this problem, the case of the present invention adopts a stratified sampling method to construct training batches. That is, first sample some categories, then sample images within these categories, and finally construct a training batch.

[0096] (4) Training process of the complete model.

[0097] First, the first stage is to use the objective function L to classify the classification model in the semantic-based hash center adaptive module. cls Perform training to ensure that the category semantic matrix W has basic category semantic information.

[0098] In the second phase:

[0099] In the space exploration training phase, the semantic-based hash center adaptive module and the binary network-based deep hash model are trained together, using the following loss function:

[0100] L2=L hc +β1L cls

[0101] Set the hyperparameter T. After T training steps, enter the code aggregation training phase and add the extrapolated aggregation loss target L. asd , and we get the following loss function:

[0102] L3=L hc +β1L cls +β2L asd

[0103] Among them, β1 and β2 are setting parameters used to control the proportion of related losses.

[0104] Considering that the specific process of model training based on the loss function can be implemented by referring to conventional technology, it will not be elaborated here.

[0105] 4. Semantic hash index construction.

[0106] After training, the deep hashing model based on binary network is deployed to a suitable device. N}Then it is scaled to the size predefined by the model, and then input into the trained binary network-based deep hash model to obtain the hash code of the candidate image set, and then the semantic hash index ψ(x) is constructed using the Multi-Index Hashing method.

[0107] 5. Online search.

[0108] When used online, the query image q input by the user is received and the image is first scaled to the size predefined by the model. Then it is input into the trained binary network-based deep hash model to obtain the corresponding hash code h q , using the trained binary network-based deep hashing model to obtain the hash code h of the query image q q , set the retrieval threshold K, and perform image retrieval in the following way:

[0109] Step (1), initializing the search radius r = 0, and the search result R;

[0110] Step (2): Search the hash code h through the semantic hash index q The image with a distance r from the search result is put into the search result R;

[0111] Step (3), calculate whether the number of images in the set R is less than the search threshold K, if so, set r = r + 1 and jump to step (2); if not, jump to step (4);

[0112] Step (4), obtaining the search result R, the images contained in it are the retrieved similar images.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0114] Embodiment 2

[0115] The present invention also provides a similar image retrieval system based on a deep hash model of a binary network, which is mainly used to implement the method provided in the above embodiment, such as Figure 4 As shown, the system mainly includes:

[0116] A model building unit, used to build a deep hashing model based on a binary network;

[0117] A model training unit is used to introduce a semantic-based hash center adaptive module to train the binary network-based deep hash model, wherein the semantic-based hash center adaptive module is provided with a classification model, and the semantic information of the category is obtained through the classification model and a hash center is generated; the training process includes two stages, in the first stage, the semantic-based hash center adaptive module is trained using the classification loss of the classification model; the second stage includes a space exploration stage and a code aggregation stage, in the space exploration stage, the hash center alignment loss is calculated using the hash center and the class binary hash representation output by the binary network-based deep hash model, and the classification loss is combined to jointly train the binary network-based deep hash model and the semantic-based hash center adaptive module; in the code aggregation stage, the class binary hash representation set is obtained using the binary network-based deep hash model, the centroid vector of each semantic category is calculated, and the extrapolation method is used for processing, and the extrapolation aggregation loss is calculated using the processing result, and the classification loss and the hash center alignment loss are combined to jointly train the binary network-based deep hash model and the semantic-based hash center adaptive module;

[0118] An image retrieval unit is used to obtain a binary hash representation set of image data in a database using a trained deep hash model based on a binary network, and to construct a semantic hash index; and to obtain a hash code of a query image using a trained deep hash model based on a binary network, and to perform image retrieval in combination with the semantic hash index.

[0119] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0120] Embodiment 3

[0121] The present invention also provides a processing device, such as Figure 5 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments.

[0122] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0123] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example:

[0124] The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.;

[0125] The output device may be a display terminal;

[0126] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

[0127] Embodiment 4

[0128] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0129] In the embodiment of the present invention, the readable storage medium is a computer-readable storage medium and can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0130] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A similar image retrieval method based on a deep hash model of a binary network, characterized in that: include: Construct a deep hashing model based on binary network; A semantic-based hash center adaptive module is introduced to train the binary network-based deep hash model, wherein a classification model is provided in the semantic-based hash center adaptive module, and semantic information of the category is obtained through the classification model and a hash center is generated; The training process includes two stages. In the first stage, the semantic-based hash center adaptive module is trained using the classification loss of the classification model. The second stage includes the space exploration stage and the code aggregation stage. In the space exploration stage, the hash center alignment loss is calculated using the hash center and the binary-like hash representation output by the deep hash model based on the binary network. The classification loss is then combined to jointly train the deep hash model based on the binary network and the semantic-based hash center adaptation module. In the code aggregation stage, the binary network-based deep hash model is used to obtain the class binary hash representation set, the centroid vector of each semantic category is calculated, and the extrapolation method is used to process it. The extrapolation aggregation loss is calculated using the processing result, and the classification loss and hash center alignment loss are combined to jointly train the binary network-based deep hash model and the semantic-based hash center adaptation module; A binary hash representation set of image data in a database is obtained using a trained binary network-based deep hash model, and a semantic hash index is constructed; and a hash code of a query image is obtained using a trained binary network-based deep hash model, and image retrieval is performed in combination with the semantic hash index.

2. According to claim 1, a similar image retrieval method based on a deep hash model of a binary network is characterized in that: The deep hash model based on the binary network includes: a binary network image feature extractor and a hash layer; wherein the image feature v is extracted by the binary network image feature extractor, and a binary-like hash representation is obtained by the hash layer. During the training phase, the class binary hash representation is used for training. After the training is completed, the class binary hash representation is converted into a binary hash code h through a symbolic function.

3. According to claim 1, a similar image retrieval method based on a deep hash model of a binary network is characterized in that: The semantic-based hash center adaptive module is provided with a classification model, and obtaining the semantic information of the category and generating the hash center through the classification model includes: The penultimate layer of the classification model is the semantic layer of the category, and the network parameters of the penultimate layer are used as the semantic information of the category, that is, the matrix W of the category semantics = [w1, w2, ..., w m ],w l is the semantic vector of the lth category, l = 1, 2, ..., m, where m represents the total number of categories; For the i-th image x i , its category information y i ∈{0, 1} m , and then perform matrix multiplication with the matrix W of category semantics to obtain the corresponding category feature w′ i : w′ i =y i W The category feature w′ is transformed into i Mapped to a probability vector p of the same dimension as the class binary hash representation i , the probability vector p i As the parameter of the multidimensional Bernoulli distribution, sampling is performed from it to obtain the hash center c′ i ={-1, 1} b , where b represents the dimension of the class binary hash representation.

4. The similar image retrieval method based on a deep hash model of a binary network according to claim 3 is characterized in that: The method of calculating the hash center alignment loss using the hash center and the binary-like hash representation output by the deep hash model based on the binary network includes: The matrix W of category semantics is used as input, and the hash center representation C = {c1, c2…, c m }, the process is as follows: [c1,c2,…,c m ] T =si g n(σ(F(W))-0.5) Among them, T is the transpose symbol, the activation function σ represents the use of each row of the matrix, c i ∈C represents the hash center representation corresponding to the lth category; The hash center alignment loss is calculated using the following formula: Among them, L hc is the hash center alignment loss, |B| is the number of images in the current training batch B, φ is the similarity calculation function, is the i-th image x i Binary hash representation of c′ i is the i-th image x i The hash center of , τ represents the temperature hyperparameter.

5. The similar image retrieval method based on a deep hash model of a binary network according to claim 1, characterized in that: The method of obtaining a binary hash representation set using a deep hash model based on a binary network, calculating a centroid vector of each semantic category, and processing by extrapolation includes: Record the current training batch data Several sets of enhanced view sets are obtained by data processing, and all enhanced view sets are used as training set B. new , the training set B is obtained through the deep hashing model based on the binary network new The binary hash representation set H new ,definition Indicates H new The class binary hash representation set belonging to category l in the class binary hash representation set H is calculated new The centroid vector of each semantic category in m represents the total number of categories; Among them, the centroid vector s corresponding to the lth category l The calculation method is: in, express The number of elements in , for An element in , i.e. a class binary hash representation; Using extrapolation method, we can obtain The new representation h′ k , expressed as: Among them, Φ represents the extrapolation method, s l for The centroid vector corresponding to the semantic category is α~U(0,1)+1, that is, α is the value sampled from U(0,1)+1, and U(0,1) represents a uniform distribution with an expected value of 0 and a variance of 1.

6. The similar image retrieval method based on a deep hash model of a binary network according to claim 5, characterized in that: The extrapolated aggregation loss is calculated using the processing results and expressed as: Among them, L asd To extrapolate the aggregation loss, for Another element in D h is the distance calculation function.

7. The similar image retrieval method based on a deep hash model of a binary network according to claim 1, characterized in that: The method of obtaining a hash code of a query image by using a trained binary network-based deep hash model and performing image retrieval in combination with a semantic hash index includes: The hash code h of the query image q is obtained using the trained binary network-based deep hashing model q , set the retrieval threshold K, and perform image retrieval in the following way: Step (1), initializing the search radius r = 0, and the search result R; Step (2), search for images with a distance r from the hash code hq through the semantic hash index, and put the retrieved images into the retrieval result R; Step (3), calculate whether the number of images in the set R is less than the search threshold K, if so, set r ==== r+1, and jump to step (2); if not, jump to step (4); Step (4), obtaining the search result R, the images contained in it are the retrieved similar images.

8. A similar image retrieval system based on a deep hash model of a binary network, characterized in that: include: A model building unit, used to build a deep hashing model based on a binary network; A model training unit, used for introducing a semantic-based hash center adaptive module to train the binary network-based deep hash model, wherein the semantic-based hash center adaptive module is provided with a classification model, and semantic information of the category is obtained through the classification model and a hash center is generated; The training process includes two stages. In the first stage, the semantic-based hash center adaptive module is trained using the classification loss of the classification model. The second stage includes the space exploration stage and the code aggregation stage. In the space exploration stage, the hash center alignment loss is calculated using the hash center and the binary-like hash representation output by the deep hash model based on the binary network. The classification loss is then combined to jointly train the deep hash model based on the binary network and the semantic-based hash center adaptation module. In the code aggregation stage, the binary network-based deep hash model is used to obtain the class binary hash representation set, the centroid vector of each semantic category is calculated, and the extrapolation method is used to process it. The extrapolation aggregation loss is calculated using the processing result, and the classification loss and hash center alignment loss are combined to jointly train the binary network-based deep hash model and the semantic-based hash center adaptation module; An image retrieval unit is used to obtain a binary hash representation set of image data in a database using a trained deep hash model based on a binary network, and to construct a semantic hash index; and to obtain a hash code of a query image using a trained deep hash model based on a binary network, and to perform image retrieval in combination with the semantic hash index.

9. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images