Depth image retrieval method and system based on metric learning and storage medium

Abstract

The invention discloses a depth image retrieval method and system based on metric learning and a storage medium, and relates to the technical field of image retrieval, a query image is input into an image hash coding model for image retrieval, and a similar image list is obtained; the training process of the image Hash coding model is as follows: constructing an image training set and a Hash center, performing feature extraction on batch images, and projecting the batch images to an embedding space to obtain feature vectors; projecting the Hash center to an embedding space to obtain a proxy; in the embedding space, the similarity between the feature vectors and the agents and between the feature vectors is optimized based on metric learning, so that the feature vectors converge to the same kind of agents, and the agents are adjusted to the same kind of feature vectors; reciprocity constraint is applied between the Hash codec and the Hash decoder, and measurement invariance constraint is applied to the Hash coding process, so that the Hash code is converged to a corresponding Hash center; according to the depth image retrieval method and system and the storage medium, the expression ability and retrieval precision of hash codes are improved.

Description

Technical Field

[0001] The present invention relates to the field of image retrieval technology, and in particular to a deep image retrieval method, system and storage medium based on metric learning. Background Art

[0002] 1) Image retrieval system based on image hashing; With the rapid development of information technology and the popularization of the Internet, massive amounts of image data are continuously generated and stored, making efficient image retrieval technology an important research direction in the fields of computer vision and artificial intelligence. In recent years, thanks to breakthroughs in deep learning, content-based image retrieval (CBIR) technology has received widespread attention and has been applied in many fields. Compared with traditional retrieval methods that rely on text tags, the CBIR system not only uses the visual features of images for similarity matching, but can also combine cross-modal information to further improve the accuracy of retrieval, providing a more intelligent search experience in application scenarios such as e-commerce, medical diagnosis, and security monitoring.

[0003] The core of content-based image retrieval lies in two key steps: calculation of feature description vector and feature matching. Figure 3 As shown in the figure. Traditional methods based on manual features can capture local features of images, but they have poor robustness in complex scenes and are difficult to adapt to the diversity of large-scale image sets. In recent years, deep neural networks have gradually become the mainstream method due to their advantages in learning high-level semantic features. They extract more discriminative features through end-to-end training and optimize retrieval efficiency in combination with index structures. Hashing methods and quantization methods are usually used to calculate feature description vectors. The hashing method maps continuous image features to discrete binary space through hash coding technology, significantly reducing storage and distance calculation costs. The quantization method reduces computational complexity by dividing high-dimensional feature vectors into finite subspaces and using lookup tables to efficiently calculate feature distances. With the continuous growth of the size of image databases and the increasing complexity of image content, how to optimize computational efficiency while ensuring retrieval accuracy remains a key issue that needs to be solved.

[0004] In image hashing, the hash function maps image features to compact binary hash codes to achieve efficient feature matching and similarity calculation. The core idea is to convert similar images into similar hash codes to speed up the indexing and comparison process. Image hashing methods can be divided into traditional hashing methods and deep learning hashing methods. Traditional hashing methods mainly rely on mathematical transformations and statistical characteristics to reduce the dimension and binarize image features, including local sensitive hashing (LSH), spectral hashing (SH), and hashing methods based on principal component analysis (PCA). These methods have low computational costs and are suitable for scenarios with more regular data distribution, but when processing complex image data, their feature expression capabilities are relatively limited, and it is difficult to ensure high retrieval accuracy.

[0005] With the rapid development of deep learning, hashing methods based on neural networks have been widely studied and have achieved superior performance on multiple tasks. Deep hashing methods usually use convolutional neural networks to extract image features and optimize hash coding in combination with specific loss functions to improve feature expression and retrieval accuracy. Usually, these methods optimize feature learning and hash coding in an end-to-end manner to improve the discriminative ability of hash codes. For example, deep neural network hashing (DNNH) uses triple ranking loss for similarity learning to optimize the distinguishing ability of hash codes; deep hashing network (HashNet) uses weighted maximum likelihood estimation to alleviate the data imbalance problem by introducing weights in the pairwise loss function and improve the robustness of the model. Compared with traditional hashing methods, deep learning hashing methods can adapt to different data distributions and automatically learn the optimal hash representation, thereby maintaining high retrieval accuracy and generalization ability in complex data environments. Deep hashing methods have been widely used in image retrieval, face recognition, anomaly detection and other fields and have achieved good results.

[0006] 2) Metric learning; Metric Learning is a machine learning method used to optimize the similarity metric between data. Its goal is to construct the optimal distance metric in a specific task so that similar samples are closer in the feature space, while dissimilar samples remain at a larger interval. Compared with traditional methods that rely on manually designed features and predefined distance functions, metric learning can adaptively adjust the structure of the feature space according to task requirements, thereby improving the performance of tasks such as classification, clustering, and retrieval. By optimizing the metric function from supervised data, this method makes the feature representation more in line with task requirements and improves the model's discriminative and generalization capabilities.

[0007] Early metric learning methods mainly rely on linear transformation and optimization strategies to learn similarity metrics between data. Among them, Mahalanobis distance learning constructs a transformation matrix to make data more separable in the new feature space, thereby optimizing the performance of nearest neighbor classification. With the development of deep learning, deep metric learning methods have become a research hotspot and have made significant progress in multiple tasks. Compared with traditional linear metric learning methods, deep metric learning relies on the powerful feature extraction ability of neural networks to learn more expressive nonlinear metric functions. Current research is mainly divided into pair-based and proxy-based methods. Pair-based methods, such as contrastive learning and twin networks, usually combine convolutional neural networks to extract image features and optimize the similarity relationship between samples through contrastive loss or triple loss. This type of method can effectively enhance the discriminative ability of feature embedding and has been widely used in tasks such as signature verification, face recognition, and fingerprint matching. However, since the number of combinations of sample pairs or triplets increases with the data size, its sample efficiency is low and the training overhead is large. In contrast, proxy-based methods optimize paired samples by learning proxies instead of pairs. For example, the proxy anchor loss directly optimizes the distance between samples and their category proxies to improve training efficiency. Such methods perform well on large-scale datasets, but due to their reliance on global proxy information, they may ignore the fine-grained information of samples and affect the characterization of complex data structures. Therefore, how to maintain the fine-grained information of data while improving training efficiency is an important challenge in current deep metric learning research.

[0008] Disadvantage 1 of the existing technology: The existing hashing methods fail to effectively utilize the advantages of metric learning. Existing image hashing methods usually directly encode image features into binary hash codes, failing to fully utilize the role of metric learning in optimizing feature representation. Introducing metric learning and directly binarizing its results may introduce quantization errors, causing similar feature vectors to be mapped to hash codes that are far apart in the hash space, thus affecting retrieval performance.

[0009] Disadvantage 2 of the existing technology: key information will be lost in the image hash coding process. Since hash coding needs to map continuous features to discrete binary space, it will inevitably lead to information loss. Especially when processing complex images, key details may not be fully preserved, and similar images will be encoded as hash codes far apart in the hash space, destroying the similarity structure of the original data and affecting the retrieval accuracy and stability.

[0010] Disadvantage 3 of the existing technology: In the existing image hash coding process, only the loss based on paired samples or the centralization loss is usually considered, resulting in limited information expression ability. The hashing method based on paired samples relies on paired supervision information, has low sample efficiency, and is difficult to fully utilize data structure information. Although the centralization loss can improve the global clustering effect, it may ignore fine-grained features, resulting in insufficient characterization of local similarities, which in turn affects the expression ability and retrieval performance of the hash code. Summary of the invention

[0011] Based on the technical problems existing in the background technology, the present invention proposes a deep image retrieval method, system and storage medium based on metric learning to improve the expression ability and retrieval accuracy of hash coding.

[0012] The deep image retrieval method based on metric learning proposed in the present invention inputs the query image into the image hash coding model for image retrieval to obtain a list of similar images; The training process of the image hash coding model is as follows: Construct an image training set and hash center, and sample batches of images from the training set each time and input them into the image hash coding model; Extract features from batch images and project them into the embedding space to obtain feature vectors, and project the hash centers into the embedding space to obtain proxies; In the embedding space, the similarity between feature vectors and agents and between feature vectors is optimized based on metric learning, so that feature vectors converge to similar agents and agents adjust to similar feature vectors. Reciprocity constraints are imposed on the hash encoder and decoder, and metric invariance constraints are imposed on the hash encoding process, so that the hash codes converge to the corresponding hash centers.

[0013] Furthermore, in optimizing the similarity between the feature vector and the agent and between the feature vectors based on metric learning, an optimization strategy based on agent-based metric learning is used to optimize the similarity between the feature vector and the agent, specifically: Optimizing the cosine similarity between the feature vector and similar agents until it exceeds a set first similarity threshold, and the optimization intensity of the cosine similarity increases as the cosine similarity decreases; The cosine similarity between the feature vector and the agents of different classes is optimized until it is lower than a set first different class similarity threshold, and the optimization intensity of the cosine similarity increases as the cosine similarity increases.

[0014] Furthermore, in optimizing the similarity between the feature vector and the agent and between the feature vectors based on metric learning, an optimization strategy based on metric learning of paired samples is adopted to optimize the similarity between the feature vectors, specifically: Optimize the cosine similarity between feature vectors of samples of the same category until it exceeds the set second similarity threshold of the same category, and the optimization intensity of the cosine similarity increases as the cosine similarity decreases; The cosine similarity between the feature vectors of samples of different categories is optimized until it is lower than the set second different-category similarity threshold, and the optimization intensity of the cosine similarity increases with the increase of the cosine similarity.

[0015] Furthermore, during the model training phase, the feature vector is modeled with a Gaussian distribution in the embedding space. Each time the optimization is performed, the feature vector is sampled from the Gaussian distribution, and then the similarity between the sampled feature vector and the agent is optimized based on metric learning.

[0016] Furthermore, the hash codec includes a hash encoder and a hash decoder, and the encoding process of the hash encoder and the decoding process of the hash decoder are constrained to be reciprocal.

[0017] Furthermore, the constrained hash coding process keeps the cosine similarity between the two hash codes after coding and the cosine similarity between the two feature vectors before coding unchanged.

[0018] Furthermore, a total loss function is constructed to train the image hash coding model. The total loss function includes the hash center loss function , metric learning loss function in embedding space And the hash encoder loss function ; Hash Center Loss Function Used to increase the Hamming distance between hash centers of different categories in the case of parameterized hash centers to meet the requirements of hash coding; Metric Learning Loss Function in Embedding Space It is used to make the feature vectors converge to similar agents in the embedding space, and the agents adjust to similar feature vectors; Hash Encoder Loss Function It is used to ensure that the encoding process is inverse to the decoding process and that the encoding process is metrically invariant.

[0019] Furthermore, during the execution of the image hash coding model, specifically: In an offline state, the image hash coding model takes an image set as input to obtain an image set hash code, wherein the image set is a collection of images that are pre-collected, stored, and used for image retrieval; In the online retrieval state, the query image is input into the image hash coding model to obtain the query image hash code; Calculate the Hamming distance between the query image hash code and each offline image hash code in the image set hash code and arrange them in ascending order. Take the offline images corresponding to the first K Hamming distances as the similar image list of the query image, where K is an integer.

[0020] A deep image retrieval system based on metric learning inputs the query image into the image hash coding model for image retrieval to obtain a list of similar images; The training process of the image hash coding model includes a training set construction module, a feature projection module, a hash center projection module and a total loss module; The training set construction module is used to construct the image training set and hash center. Each time, a batch of images are sampled from the training set and input into the image hash coding model. The feature projection module is used to extract features from batch images and project them into the embedding space to obtain feature vectors; The hash center projection module is used to project the hash center into the embedding space to obtain the proxy; The total loss module is used to optimize the similarity between feature vectors and agents in the embedding space based on metric learning, so that feature vectors converge to similar agents and agents adjust to similar feature vectors; at the same time, reciprocity constraints are imposed between hash encoders and decoders, and metric invariance constraints are imposed on the hash encoding process, so that the hash code converges to the corresponding hash center.

[0021] A computer-readable storage medium, characterized in that a plurality of classification programs are stored on the computer-readable storage medium, and the plurality of classification programs are used to be called by a processor and execute the depth image retrieval method as described above.

[0022] The advantages of the deep image retrieval method, system and storage medium based on metric learning provided by the present invention are: introducing agent-based metric learning in the deep hash system to enhance the feature representation capability, and combining it with metric learning based on paired samples to effectively capture fine-grained information; hash coding processing is realized by constructing a hash codec (hash encoder and hash decoder), and the encoding process of the hash encoder has the invariance of cosine similarity to ensure the discrimination ability and consistency of the hash coding, and to ensure that the hash code converges to the corresponding hash center, thereby optimizing the stability and distinguishability of the hash mapping; a reversible mapping between feature vectors in the embedded space and hash codes in the Hamming space is realized through the hash codec to reduce the loss of information in the encoding process and improve the expression ability and retrieval accuracy of the hash coding. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 Training flow chart for image hash encoding model; Figure 3 Schematic diagram of existing content-based image retrieval methods. DETAILED DESCRIPTION

[0024] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0025] like Figures 1 to 3 As shown, the deep image retrieval method based on metric learning proposed by the present invention inputs the query image into the image hash coding model for image retrieval to obtain a similar image list; The training process of the image hash coding model is as follows: Step 1: Extract features from batch images and project them into the embedding space to obtain feature vectors, and project the hash centers into the embedding space to obtain proxies; Step 2: In the embedding space, the similarity between feature vectors and agents and between feature vectors is optimized based on metric learning, so that feature vectors converge to similar agents and agents adjust to similar feature vectors; Step 3: impose reciprocity constraints on the hash codec and metric invariance constraints on the hash encoding process, so that the hash code converges to the corresponding hash center.

[0026] Through steps one to three, this embodiment introduces proxy-based metric learning in the deep hash system to enhance the feature representation capability, and combines it with paired sample-based metric learning to effectively capture fine-grained information; hash coding processing is implemented by constructing a hash codec (hash encoder and hash decoder), and the hash encoder keeps the cosine similarity between the two hash codes after encoding and the cosine similarity between the two feature vectors before encoding unchanged during the encoding process to ensure the discrimination ability and consistency of the hash coding, and to ensure that the hash code converges to the corresponding hash center, thereby optimizing the stability and distinguishability of the hash mapping; a reversible mapping between the feature vector in the embedding space and the hash code in the Hamming space is implemented through the hash codec to reduce the loss of information during the encoding process and improve the expression ability and retrieval accuracy of the hash coding.

[0027] In one of the embodiments, a backbone network and a feature encoder are used to extract and project features of an image. Specifically, the backbone network is used to encode the image into image features represented by vectors, and the feature encoder is used to project the image features into an embedding space. The backbone network and the feature encoder may adopt an existing network structure for image processing, which will not be described in detail in this embodiment.

[0028] In this embodiment, the construction process of the hash center is as follows: In the image hashing task, the hash center is used to represent the binary hash code of a certain category of images. There are two ways to construct the hash center: predefined hash center and parameterized hash center.

[0029] The predefined hash centers are designed to make the Hamming distance between different categories as large as possible. When the length of the hash code is a power of 2 and the number of image categories does not exceed twice the length of the hash code, the Hadamard matrix can be used for construction. Specifically, if the hash code length is The number of categories is , we can choose the Hadamard matrix and Front after vertical splicing If the relationship between the length of the hash code and the number of categories does not meet the above conditions, the hash center can be generated by random sampling, that is, from the length of Random sampling from a Bernoulli distribution with parameter 0.5 times, and use the sampling result as the hash center.

[0030] The parameterized hash center uses a trainable vector to represent the hash center of each category. During the training process, the image hash coding model optimizes the hash center by minimizing the metric learning loss and quantization loss to increase the Hamming distance between hash centers of different categories, while constraining the binarization of the hash center to meet the requirements of hash coding. The parameterized hash center can be dynamically adjusted according to the data distribution, thereby improving the effect of hash learning.

[0031] The metric learning loss of the hash center uses the first proxy anchor loss function : ; (1) in, represents the cosine similarity of two vectors, represents a threshold value, is a scaling factor, represents the set of hash centers, Represents the set of positive hash centers of the data in the batch image. The positive hash center is the set of hash centers with the same class label as the batch image. represents the hash code of the batch of images, is the hash code of the image, is divided into two sets, For hash center A collection of similar hash codes, Hash Center A collection of different types of hash codes.

[0032] Quantization loss function of hash center ,in An index used to distinguish different loss functions, which has no actual physical meaning: ; (2) in, A hash code representing a batch of images.

[0033] Total hash center loss function for: ; (3) in, is a weight coefficient used to balance the metric learning loss and the quantization loss.

[0034] In one embodiment, before optimizing the similarity between the feature vector and the proxy based on metric learning, Gaussian distribution modeling is performed on the feature vector to generate a feature representation in an embedding space, and then the similarity between the feature representation and the proxy is optimized based on metric learning. The feature vector in the embedding space is modeled as a Gaussian distribution to characterize the uncertainty in the image feature representation, thereby improving the robustness of the hash code.

[0035] Specifically, the feature encoder is first used to project the image features into the embedding space and model them based on the Gaussian distribution, where the output of the feature encoder includes the mean vector and the logarithmic variance vector. In the training phase, feature vectors are sampled from the Gaussian distribution for feature encoding, and the reparameterization method is used to implement gradient backpropagation, that is, the feature vector is represented by the product of the standard deviation vector and the noise sampled from the standard Gaussian distribution plus the mean vector. The feature vector is modeled with the Gaussian distribution in the embedding space. Each time the feature vector is optimized, it is sampled from the Gaussian distribution, and then the similarity between the sampled feature vector and the agent is optimized based on metric learning. At the same time, the hash center is mapped using a hash decoder to obtain the agent in the embedding space, thereby optimizing the similarity between the feature vector and the agent based on metric learning. In the testing phase, the mean vector is directly used as the feature vector to reduce the impact of randomness and ensure the stability of feature encoding.

[0036] In one of the embodiments, the similarity between feature vectors and agents and between feature vectors is optimized based on metric learning. Specifically, in the process of optimizing the embedding space feature encoding, two optimization strategies, agent-based metric learning and paired sample-based metric learning, are adopted.

[0037] (a1) Optimize the similarity between feature vectors and agents using the optimization strategy of agent-based metric learning: Agent-based metric learning constructs agents and optimizes the similarity between sample feature vectors and agents, so that samples of the same category are closer in the embedding space, while the discrimination between samples of different categories is improved. Finally, the feature vector converges to the corresponding agent, achieving global optimization and improving the discriminability and optimization efficiency of feature representation.

[0038] Second Proxy Anchor Loss Function for: ; (4) in, represents the cosine similarity of two vectors, represents a distance threshold, is a scaling factor, represents a collection of agents, represents the set of positive proxies for the data in the batch of images, represents the feature vector of a batch of images, is divided into two sets, For Agents A collection of similar eigenvectors, For Agent A collection of feature vectors of different classes.

[0039] It should be noted that the agents in the embedding space are all calculated by the hash center through the hash decoder, and the positive agent is the set of agents with the same class label as the batch image.

[0040] For ease of understanding, the loss function It can be rewritten as follows: ; (5) in, , is a smooth approximation of the activation function ReLU, parameter Refers to the above formula (4) or , , For collection or Collection The total number of eigenvectors in , Log-Sum-Exp is a smooth approximation of the maximum function. Refers to the above formula (4) or .

[0041] Based on formulas (4) and (5), the optimization strategy of agent-based metric learning is directly quantified to optimize the similarity between feature vectors and agents. Specifically, the cosine similarity between feature vectors and agents of the same type is optimized until it exceeds the set first similarity threshold of the same type, and the optimization intensity of cosine similarity increases as the cosine similarity decreases; the cosine similarity between feature vectors and agents of different types is optimized until it is lower than the set first similarity threshold of different types, and the optimization intensity of cosine similarity increases as the cosine similarity increases.

[0042] (a2) Optimize the similarity between feature vectors using an optimization strategy based on metric learning of paired samples: Metric learning based on paired samples optimizes the similarity relationship between features of sample pairs, so that the feature similarity between samples of the same category is improved and the similarity between samples of different categories is reduced, thereby improving the discriminative ability of feature encoding.

[0043] Paired Sample Loss Function for: ; (6) in, is the feature vector of the batch image, The eigenvector is removed A collection of is the feature vector The positive sample set (excluding ), , is the feature vector The positive sample set is a set of feature vectors of sample images of the same category, and the negative sample set is a set of feature vectors of sample images of different categories.

[0044] Similarly, formula (6) can also be written as In the form of, the above paired sample loss function It can also be replaced by the existing triplet loss to construct a metric learning loss function based on paired samples.

[0045] Based on formula (6), the similarity between feature vectors is optimized by directly quantifying the optimization strategy based on metric learning of paired samples. Specifically, the cosine similarity between feature vectors of samples of the same category is optimized until it exceeds the set second similarity threshold of the same category, and the optimization intensity of cosine similarity increases as the cosine similarity decreases; the cosine similarity between feature vectors of samples of different categories is optimized until it is lower than the set second similarity threshold of different categories, and the optimization intensity of cosine similarity increases as the cosine similarity increases.

[0046] The optimization strategy of using metric learning based on paired samples relies on a large number of sample pairs, and the optimization efficiency is relatively low, but it can capture more detailed feature information and optimize the local structure of the feature space. The combination of the above two metric learning optimization strategies (agent-based metric learning and paired sample-based metric learning) can further improve the overall discrimination of the feature space while maintaining the optimization efficiency.

[0047] Therefore, the metric learning loss function in the embedding space is for: ; (7) in, is a weight coefficient used to balance the proxy anchor loss and paired sample loss.

[0048] The smooth activation function in formula (4) and (6) can improve the stability of gradient calculation, and the approximate maximum function therein can better optimize the metric relationship. The optimization process is calculated based on batch data to improve training efficiency and enhance feature representation capabilities. Through the above two optimization strategies, the consistency and distinguishing ability of feature mapping are improved, thereby improving the accuracy of subsequent hash coding. Compared with the prior art, the optimized hash space feature distribution of this embodiment is more reasonable, making the feature representation of the same category of data more compact, while improving the distinguishability of different categories of data.

[0049] In one embodiment, a hash codec is set to implement the mapping conversion between the feature vector of the embedding space and the hash code of the Hamming space. The hash codec includes a hash encoder and a hash decoder.

[0050] The hash encoding and decoding process must satisfy the reciprocity of the encoding process and the decoding process, and ensure that the encoding process maintains the invariance of the cosine similarity, that is, the constrained hash encoding process maintains the cosine similarity between the two hash codes after encoding and the cosine similarity between the two feature vectors before encoding. To this end, the hash codec can be implemented based on matrix operations, where the decoding matrix is ​​the transpose of the encoding matrix, and the product of the transpose of the encoding matrix and the encoding matrix approaches the unit matrix, while the product of the encoding matrix and its transpose also approaches the unit matrix, thereby ensuring that the encoding and decoding processes are reciprocal and that the encoding process maintains the metric relationship of the embedded spatial features.

[0051] Specifically, the hash coding matrix is , the hash decoding matrix is , the loss function of hash coding : ; (8) in, represents the identity matrix, is the F-norm (Frobenius norm) of the matrix.

[0052] Through the setting of hash codec, the proxy corresponds to the hash center one by one. Therefore, this embodiment optimizes the hash encoder parameters based on the reciprocity of the hash encoder and the hash decoder and the consistency of the cosine similarity to ensure that the hash code has a higher similarity with the hash center of the same type, so that the feature vector in the embedding space converges to the corresponding hash center after being encoded by the hash encoder, and generates a hash code with good discrimination, thereby improving the stability and retrieval performance of the hash mapping. Image hashing based on metric learning in hash code embedding space Figure 2 As shown in Figure 2. The hash decoder maps the hash center to the proxy in the embedding space and uses metric learning to keep the feature vectors of the same type of image at a small distance from their corresponding proxy. The hash encoder maintains the consistency of cosine similarity during the encoding process, thereby ensuring that the distance between the generated hash code and its corresponding hash center is small enough to improve the accuracy and stability of hash coding. In addition, the hash encoder-decoder ensures the recoverability of the encoding process and effectively reduces information loss, thereby improving the quality of hash coding and retrieval performance.

[0053] In this embodiment, a total loss function is constructed to train the image hash coding model. The total loss function includes a hash center loss function. , metric learning loss function in embedding space And the hash encoder loss function ; Hash Center Loss Function Used to increase the Hamming distance between hash centers of different categories in the case of parameterized hash centers to meet the requirements of hash coding; metric learning loss function in embedding space Used to make feature vectors converge to similar agents in the embedding space, and agents adjust to similar feature vectors; hash encoder loss function It is used to ensure that the encoding process is inverse to the decoding process and that the encoding process is metric invariant, so that the hash code converges to the same hash center. The model parameters in the image hash coding model are optimized based on the total loss function and using stochastic gradient descent.

[0054] After the image hash coding model is trained, it enters the execution phase. The deep image retrieval process of online images is as follows (b1) to (b3): (b1) In an offline state, the image hash coding model takes an image set as input to obtain an image set hash code. The image set is a collection of images that are pre-collected, stored, and used for image retrieval and is usually not updated in real time. (b2) In the online retrieval state, the query image is input into the image hash coding model to obtain the query image hash code; (b3) Calculate the Hamming distance between the query image hash code and the hash code of each offline image in the image set and arrange them in ascending order. Take the offline images corresponding to the first K Hamming distances as the similar image list of the query image, where K is an integer.

[0055] In (b1) and (b2), the image set hash code and the query image hash code are both binary hash code structures, specifically using the symbolic function The output of the hash encoder is converted into a binary hash code of -1 and 1. By using binary hash codes, the system's storage overhead and distance calculation overhead can be effectively reduced, and the efficiency of the image retrieval system can be improved.

[0056] This embodiment proposes a deep image retrieval system based on metric learning, which inputs the query image into the image hash coding model for image retrieval to obtain a list of similar images; The training process of the image hash coding model includes a training set construction module, a feature projection module, a hash center projection module and a total loss module; The training set construction module is used to construct the image training set and hash center. Each time, a batch of images are sampled from the training set and input into the image hash coding model. The feature projection module is used to extract features from the image and project them into the embedding space to obtain feature vectors; The hash center projection module is used to project the hash center into the embedding space to obtain the proxy; The total loss module is used to optimize the similarity between feature vectors and agents in the embedding space based on metric learning, so that feature vectors converge to similar agents and agents adjust to similar feature vectors; at the same time, reciprocity constraints are imposed between hash encoders and decoders, and metric invariance constraints are imposed on the hash encoding process, so that the hash code converges to the corresponding hash center.

[0057] In addition, a computer-readable storage medium is proposed, on which a plurality of classification programs are stored. The plurality of classification programs are used to be called by a processor and execute the depth image retrieval method as described above.

[0058] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0059] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A deep image retrieval method based on metric learning, characterized in that: Input the query image into the image hash coding model for image retrieval to obtain a list of similar images; The training process of the image hash coding model is as follows: Construct an image training set and hash center, and sample batches of images from the training set each time and input them into the image hash coding model; Extract features from batch images and project them into the embedding space to obtain feature vectors, and project the hash centers into the embedding space to obtain proxies; In the embedding space, the similarity between feature vectors and agents and between feature vectors is optimized based on metric learning, so that feature vectors converge to similar agents and agents adjust to similar feature vectors. Reciprocity constraints are imposed on the hash encoder and decoder, and metric invariance constraints are imposed on the hash encoding process, so that the hash codes converge to the corresponding hash centers.

2. The deep image retrieval method based on metric learning according to claim 1, characterized in that: In optimizing the similarity between feature vectors and agents and between feature vectors based on metric learning, an optimization strategy based on agent-based metric learning is used to optimize the similarity between feature vectors and agents, specifically: Optimizing the cosine similarity between the feature vector and similar agents until it exceeds a set first similarity threshold, and the optimization intensity of the cosine similarity increases as the cosine similarity decreases; The cosine similarity between the feature vector and the agents of different classes is optimized until it is lower than a set first different class similarity threshold, and the optimization intensity of the cosine similarity increases as the cosine similarity increases.

3. The deep image retrieval method based on metric learning according to claim 1, characterized in that: In optimizing the similarity between feature vectors and agents and between feature vectors based on metric learning, an optimization strategy based on metric learning of paired samples is used to optimize the similarity between feature vectors, specifically: Optimize the cosine similarity between feature vectors of samples of the same category until it exceeds the set second similarity threshold of the same category, and the optimization intensity of the cosine similarity increases as the cosine similarity decreases; The cosine similarity between the feature vectors of samples of different categories is optimized until it is lower than the set second different-category similarity threshold, and the optimization intensity of the cosine similarity increases with the increase of the cosine similarity.

4. The deep image retrieval method based on metric learning according to claim 1, characterized in that: During the model training phase, the feature vector is modeled with a Gaussian distribution in the embedding space. Each time the optimization is performed, the feature vector is sampled from the Gaussian distribution, and then the similarity between the sampled feature vector and the agent is optimized based on metric learning.

5. The deep image retrieval method based on metric learning according to claim 1, characterized in that: The hash codec includes a hash encoder and a hash decoder, and the encoding process of the hash encoder and the decoding process of the hash decoder are constrained to be reciprocal.

6. The deep image retrieval method based on metric learning according to claim 1, characterized in that: The constrained hash coding process keeps the cosine similarity between the two hash codes after encoding and the cosine similarity between the two feature vectors before encoding unchanged.

7. The deep image retrieval method based on metric learning according to claim 1, characterized in that: Construct a total loss function to train the image hash coding model. The total loss function includes the hash center loss function. , metric learning loss function in embedding space And the hash encoder loss function ; Hash Center Loss Function Used to increase the Hamming distance between hash centers of different categories in the case of parameterized hash centers to meet the requirements of hash coding; Metric Learning Loss Function in Embedding Space It is used to make the feature vectors converge to similar agents in the embedding space, and the agents adjust to similar feature vectors; Hash Encoder Loss Function It is used to ensure that the encoding process is inverse to the decoding process and that the encoding process is metrically invariant.

8. The deep image retrieval method based on metric learning according to claim 1, characterized in that: During the execution of the image hash coding model, specifically: In an offline state, the image hash coding model takes an image set as input to obtain an image set hash code, wherein the image set is a collection of images that are pre-collected, stored, and used for image retrieval; In the online retrieval state, the query image is input into the image hash coding model to obtain the query image hash code; Calculate the Hamming distance between the query image hash code and each offline image hash code in the image set hash code and arrange them in ascending order. Take the offline images corresponding to the first K Hamming distances as the similar image list of the query image, where K is an integer.

9. A deep image retrieval system based on metric learning, characterized in that: Input the query image into the image hash coding model for image retrieval to obtain a list of similar images; The training process of the image hash coding model includes a training set construction module, a feature projection module, a hash center projection module and a total loss module; The training set construction module is used to construct the image training set and hash center. Each time, a batch of images are sampled from the training set and input into the image hash coding model. The feature projection module is used to extract features from batch images and project them into the embedding space to obtain feature vectors; The hash center projection module is used to project the hash center into the embedding space to obtain the proxy; The total loss module is used to optimize the similarity between feature vectors and agents in the embedding space based on metric learning, so that feature vectors converge to similar agents and agents adjust to similar feature vectors; at the same time, reciprocity constraints are imposed between hash encoders and decoders, and metric invariance constraints are imposed on the hash encoding process, so that the hash code converges to the corresponding hash center.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of classification programs, and the plurality of classification programs are used to be called by a processor and execute the depth image retrieval method as described in any one of claims 1 to 8.

Images