Deep hash retrieval method based on meta-learning

Abstract

The present invention discloses a deep hash retrieval algorithm based on meta-learning, 1) pre-processing image data to convert a balanced image data set into an unbalanced image data set; 2) updating the parameters of the meta-learning network and the parameters of the image retrieval network; optimizing in the image retrieval network using the unbalanced image data set, and optimizing in the meta-learning network using the balanced image data set; 3) constructing a learning method for the image retrieval training process; 4) updating the parameters in the image retrieval training process; 5) in the image retrieval generation hash code process, the input image is activated as a visual feature through the feature activation of the fully connected layer of the convolutional neural network CNN, and the visual feature is used for small-scale image retrieval. Through the weighting function from the training loss to the sample weight, the algorithm does not need to manually pre-specify the weight function and additional hyperparameters, and can directly learn the explicit weighting function adaptively from the data.

Description

Technical Field

[0001] The present invention relates to the field of image retrieval technology, and in particular to a deep hash retrieval method based on meta-learning to process the problem of image data imbalance. Background Art

[0002] In the past few decades, the rapid development of the Internet has ushered in a new era for mankind. In particular, the rapid development of mobile Internet has seen more and more users begin to surf the Internet with their mobile phones or laptops, which also shows that mobile Internet is gradually penetrating into all areas of people's lives and work. There are many sources of network data, such as weather sensors, social media websites, online banking and mobile phone signals. The rapid development of a variety of mobile Internet applications such as WeChat, Alipay, and location services is profoundly changing social life in the information age. In recent years, it has achieved a leapfrog development from 3G to 4G to 5G. With the rapid development of the Internet industry, the data accumulated in all walks of life have shown an explosive growth trend. These data include not only text, but also images, audio, and video. According to statistics, Internet users upload more than 1 billion pictures on the Internet every day; Taobao members upload more than 100 million pictures every day; Facebook has more than 1 billion registered users and uploads more than 1 billion pictures every month. These fully prove that a large amount of data is generated in all walks of life, and mankind has entered the "big data" era. In this era, how users can efficiently and accurately retrieve the required information from massive, high-dimensional data has become a key issue to be faced.

[0003] For small and medium-sized image databases, traditional nearest neighbor image retrieval is generally used. In order to improve the accuracy of retrieval, the dimension d of image features can reach tens of thousands of dimensions. When facing large-scale image databases, it will cause two problems: (1) The workload of retrieving nearest neighbor images is heavy; (2) A huge storage space is required to store image features. The above will lead to problems such as large image feature storage space and slow retrieval speed. Therefore, researchers have proposed an "approximate nearest neighbor" retrieval method to solve the problem of large-scale image feature storage and fast retrieval. "Approximate nearest neighbor" samples refer to samples whose distance from the query sample is d times the distance between the "nearest neighbor" sample and the query sample. d>1 is called the approximation factor. When facing a large-scale data set, the computational cost of searching for "approximate nearest neighbor" samples is obviously smaller than that of linear retrieval of "nearest neighbor" samples. Among them, the algorithm based on image hashing is one of the most effective methods for approximate nearest neighbor retrieval. Traditional hashing algorithms mainly map high-dimensional data to low-dimensional Hamming space through hash functions, and then use Hamming distance to represent the similarity measure.

[0004] In recent years, hash algorithms have been widely used in image retrieval. Existing methods can be roughly divided into two categories: data-independent and data-dependent hash algorithms. Data-independent methods mainly rely on random projections to construct hash functions. Locality-sensitive hashing (LSH) is a typical method that uses random linear projections to map similar data to similar binary codes. However, data-independent hash algorithms require longer hash codes to achieve higher accuracy. Data-dependent hash algorithms learn the projection matrix from the original data, which can effectively reduce the length of the hash code.

[0005] In recent years, deep learning has also been applied to large-scale image hashing algorithms. To achieve better retrieval performance, deep learning-based hashing algorithms require a large and high-quality data set to train the model. However, in real life, most data sets have data bias. The three most typical biases are: 1) category imbalance, that is, a very small number of categories are easy to collect, but most categories are difficult to collect; 2) data noise, the data itself is noisy; 3) label noise: because the cost of obtaining labels is too high, a simple network search is performed to annotate the data, so there will be many wrong labels, which will also cause difficulties in training.

[0006] Among them, unbalanced data is ubiquitous in real life. For real data, the amount of data in different categories is generally not ideally evenly distributed, but often unbalanced, such as cancer patient diagnosis, bankruptcy prediction, and credit card fraud detection. If the category distribution in the data set is extremely unbalanced, reaching the point where the number of samples in one or more data categories is far greater than the number of samples in another or more data categories, such data is called category unbalanced data. These unbalanced data can be difficult to analyze because machine learning algorithms tend to show biases in the majority of classes, leading to misleading conclusions. Also in medical image analysis, searching for similar images (in terms of similar anatomical structures) for diagnostic purposes can act as a "virtual peer review." Retrieving similar images from archives of past cases can be very helpful for diagnostic images. However, most existing real data sets have the characteristics of large numbers and unbalanced data categories. This makes it very difficult to retrieve valuable and meaningful information from these data, and it is a challenging topic for relevant researchers.

[0007] To address the problem of imbalanced data sets, this paper proposes a deep hash retrieval algorithm based on meta-learning, which can effectively improve the robustness of the model and reduce the impact of imbalanced data on the efficiency of image retrieval.

[0008] Meta-learning, also known as learning to learn, literally means learning how to learn. From the literal meaning, it seems that this type of algorithm is closer to the way humans learn - drawing inferences from one instance and applying it to other instances. Good machine learning models usually require a large number of samples for training. In contrast, humans can learn new concepts and skills faster and more effectively. For example, a child who has only seen cats and birds a few times can quickly distinguish them; a person who can ride a bicycle can quickly master a motorcycle. Meta-learning aims to train models that can learn new skills or quickly adapt to new environments through a small number of training samples. The birth of meta-learning has prompted machine learning to advance to another side, realizing artificial intelligence in a way that is closer to humans and more efficient. Summary of the invention

[0009] In order to solve the problem of unbalanced datasets in image retrieval, the purpose of the present invention is to provide a deep hash retrieval method based on meta-learning. Through the weighting function from training loss to sample weights, the algorithm does not need to manually pre-specify the weight function and additional hyperparameters, and can adaptively learn explicit weighting functions directly from data.

[0010] The deep hash retrieval method based on meta-learning of the present invention is special in that it includes the following steps:

[0011] 1) Preprocess the image data and convert the balanced image data set into an unbalanced image data set;

[0012] 2) Parameters of the meta-learning network , parameters of the image retrieval network Make updates;

[0013] Using unbalanced image datasets in image retrieval networks Optimize by minimizing the loss of the unbalanced image dataset and reweighting the samples to obtain the optimal ; Using balanced image datasets in meta-learning networks Optimize, through the optimization of the image retrieval network To optimize , get the optimal ;

[0014] 3) Construct a learning method for the image retrieval training process, divide the image retrieval network into multiple batches to iteratively train the images in the unbalanced image dataset, obtain the update equations of the image retrieval network parameters under the current number of iterative training times for multiple batches, and calculate the update of the meta-learning network parameters based on the feedback obtained from the updated parameters of the image retrieval network;

[0015] 4) Update the parameters in the image retrieval training process by updating the parameters To improve the parameters during model training ;

[0016] 5) Image retrieval generates hash code process. The input image is activated by the fully connected layer of the convolutional neural network CNN as visual features, and the visual features are used for small-scale image retrieval.

[0017] Furthermore, data of the same category form a category sample, and the balanced image dataset is that the number of samples in each category is equal among samples of multiple categories; the unbalanced image dataset is that the number of samples in each category is different among samples of multiple categories.

[0018] Furthermore, we use the unbalanced image dataset to perform To optimize, define , Indicates samples, It is The label vector of the class, N is the total number of unbalanced image datasets, and the value of N is a natural number. represents the classifier, represents the classifier parameters;

[0019] 1) Utilize Calculate the minimization loss for the unbalanced image dataset and Recorded as , For the unbalanced image dataset Sample loss;

[0020] 2) When training an unbalanced image dataset, use a weighted network Perform sample reweighting to improve training robustness, represents the meta-learning network parameters;

[0021] 3) Optimize by minimizing loss and reweighting samples Get the best ;

[0022] (1).

[0023] Furthermore, balancing image data represents the distribution of correctly labeled samples, Represents the first samples, Representative The label information of samples, and , It is the number of samples in the balanced image dataset in the meta-learning network, obtained through formula (1). to optimize ;

[0024] (2),

[0025] where , is calculated from the metadata.

[0026] Furthermore, a learning method for constructing the image retrieval training process is to divide multiple batches in the image retrieval network to iteratively train the images in the unbalanced image dataset, where each batch is where is the batch size, n < N; then, each batch uses formula (1) to obtain the direction of the target loss L decrease to move the current , t is the number of iterations, representing the number of update times, and the update equation of the image retrieval network parameters is obtained:

[0027] (3);

[0028] According to the feedback obtained from the updated parameters of the image retrieval network, the update of the meta-learning network parameters can be calculated:

[0029] (4);

[0030] where is the step size, m ≤ M.

[0031] Furthermore, the parameters in the image retrieval training process are updated, and the parameters are updated to improve the parameters in the model training process;

[0032] (5).

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0034] Existing hashing methods usually only focus on how to maintain the similarity learning between samples, ignoring the problem of data class imbalance in the dataset, which may reduce the retrieval performance. Through a large number of experiments, it is proved that the deep hashing retrieval algorithm based on meta-learning proposed by the present invention can effectively reduce the impact of unbalanced data on image retrieval.

[0035] The algorithm of this application can automatically learn a weighting function directly from the data. The weighting function is a multi-layer perceptron with only one hidden layer. Under the guidance of a small amount of unbiased metadata, the parameters of the weighting function can be optimized and updated simultaneously with the parameters in the model training process. The update equation of the meta-learning network parameters can be interpreted as: the weight of samples that are more consistent with the meta-learning data will be increased, and the weight of samples that do not meet the meta-learning data will be reduced. The deep hash retrieval algorithm based on meta-learning can effectively reduce the impact of unbalanced data on image retrieval and improve the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Meta-learning weight network.

[0037] Figure 2 Flowchart of meta-learning branches in image retrieval.

[0038] Figure 3 For the image retrieval process.

[0039] Figure 4 Comparison of mAP@all for four code lengths under balanced dataset.

[0040] Figure 5 This is the mAP@all comparison of 32-bit code length under unbalanced dataset. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the following Figure 1-5 The present invention is further described in detail. Example

[0042] The present invention proposes a deep hash retrieval method based on meta-learning, which uses meta-learning to automatically learn a weight loss function from data. Due to the universal approximation ability of the weight network, it can fit the weight function well. The algorithm is divided into two parallel networks. One is the meta-learning network model, and the other is the image retrieval network model. Under the guidance of a small amount of indifferent meta-learning data, the meta-learning data here refers to a part of balanced data extracted from the image data as a training set, so that the parameters of the weighting function can be optimized and updated simultaneously with the parameters of the model training. The algorithm can effectively reduce the impact of the category imbalance problem on the accuracy of image retrieval.

[0043] The deep hash retrieval method based on meta-learning of the present invention comprises the following steps:

[0044] The image data is preprocessed to convert a balanced image data set into an unbalanced image data set; in the present application, data of the same category form a category sample, and the balanced image data set is that the number of samples in each category is equal among samples of multiple categories; the unbalanced image data set is that the number of samples in each category is different among samples of multiple categories.

[0045] For example, a balanced image dataset is a dataset with 10 categories of images, namely airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. The number of samples in each category is equal, so the dataset is called a balanced image dataset. An unbalanced dataset is a dataset with 10 categories of images, and there is a certain deviation in the number of samples in each category. Suppose the number of airplane data is 5000, the number of car data is 2000, and so on... The number of truck data is 200, then this dataset is called an unbalanced image dataset.

[0046] Then the parameters of the meta-learning network are , parameters of the image retrieval network To update:

[0047] Using unbalanced image datasets in image retrieval networks Optimize and set the training set ,in Indicates samples, It is The label vector of the class. This application takes the binary classification problem as an example. There are two categories of samples. If y=0, it is the first category sample, c is 1; if y=1, it is the second category sample, and c is 2. N is the total number of unbalanced image data sets, and the value of N is a natural number. represents the classifier, Represents the classifier parameters, x takes the value ;

[0048] 1) Utilize Calculate the minimization loss for the unbalanced image dataset and Recorded as , For the unbalanced image dataset Sample loss;

[0049] 2) When training an unbalanced image dataset, use a weighted network Perform sample reweighting to improve training robustness, represents the meta-learning network parameters;

[0050] 3) Optimize by minimizing loss and reweighting samples Get the best ;

[0051] (1).

[0052] Using balanced image datasets in meta-learning networks Optimize and balance image data represents the distribution of correctly labeled samples, Represents the first samples, Representative The label information of samples, and , is the number of balanced image dataset samples in the meta-learning network, obtained by formula (1) To optimize ;

[0053] (2),

[0054] in , Calculated from metadata;

[0055] For this application Guided Optimization , in turn through Adjustment , the two take advantage of each other.

[0056] Construct a learning method for the image retrieval training process. The training process adopts an iterative method. In the image retrieval network, multiple batches are divided to iteratively train the images in the unbalanced image dataset. The multiple batches of each training are ,in is the batch size; then, by a complete training on multiple small batches of training data, we follow the formula , get the direction of the target loss L to move the current , we get the update equation of the image retrieval network parameters:

[0057] (3);

[0058] Among them, t is the number of iterations, which also represents the number of updates.

[0059] Based on the feedback obtained from the parameters updated by the above network, the metadata can be calculated:

[0060] (4);

[0061] in, is the step size, m≤M.

[0062] Update the parameters in the image retrieval training process by updating the parameters To improve the parameters during model training ;

[0063] (5).

[0064] Image retrieval generates hash codes. CNN refers to Convolutional Neural Network. Its basic structure consists of input layer, pooling layer, fully connected layer and output layer. Layers 6 to 8 here refer to fully connected layers. The input image is activated as visual features through the features of layers 6 to 8 of CNN, and visual features are used for small-scale image retrieval. This feature achieves good performance in small-scale image retrieval and categories. When faced with large-scale data, since this feature is a high-dimensional vector, it will greatly affect the efficiency and performance of retrieval. Researchers have proposed a method to convert feature vectors into binary codes, which can reduce computational costs and storage space. After conversion to binary codes, Hamming distance or hash codes can be used for fast comparison.

[0065] In this embodiment, more specifically, the present application adjusts the number of samples in the balanced image data set to an unbalanced image data set with a difference in the number of samples by selecting samples during conversion. The formation of the unbalanced data set of the present application preprocesses the image data of the three balanced data sets of CIFAR-10, CIFAR-100 and STL-10, and converts the balanced data set into unbalanced image data with a long-tail distribution. Taking CIFAR-10 as an example, the CIFAR-10 data set is a single-label data set for identifying universal objects, and contains a total of 10 categories of RGB color pictures: airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships and trucks. The number of samples in each category is 5000. CIFAR-10 is preprocessed, assuming that the number of airplanes is still 5000, the number of samples of cars is 2000, the number of samples of birds is 1000, and so on. The number of samples of trucks is 200, so that an unbalanced image data set is formed.

[0066] For the meta-learning network method, the goal of this algorithm is to automatically learn hyperparameters during the learning process To achieve this goal, As a multilayer perceptron with only one hidden layer, the multilayer perceptron contains 100 nodes, such as Figure 1As shown, this weight network is called a meta-learning weight network. Each hidden node uses a Relu activation function, and the output uses a Sigmoid activation function to ensure that the output is in the interval [0,1]. The weight network is considered to be a universal approximator of almost any continuous function, so it can adapt to a wide range of weight functions, including those used in traditional research.

[0067] For meta-learning networks in image retrieval: computing the optimal and Two nested loop optimizations are required, here a single optimization loop is used to exploit online policy updates and , to ensure the efficiency of the algorithm.

[0068] Update the parameters of the meta-learning network based on the feedback obtained from the updated parameters of the classifier, such as Figure 2 Parameters of the meta-learning network shown It can be calculated by formula 2. Then, by updating the parameters of the meta-learning network To improve the parameters during model training .

[0069] For image retrieval, the hash code generation process is as follows: Figure 3 As shown, assuming The final output of depends on a set of hidden layers, which produce images with similar binary codes with the same label. and A hidden layer H is embedded between them. The hidden layer is a fully connected layer whose neuron activity is determined by the subsequent encoding semantics and the realization of the category. The proposed hidden layer not only provides The abstraction of rich features and the connection between mid-level features and high-level semantics. In the present invention, the neurons in the hidden layer are activated by the Sigmoid function, so the activation is approximately .

[0070] For image I, we first extract the output of the hidden layer and set it to Then, the binary code is obtained by binarization through threshold activation. ( represents the number of nodes in the hidden layer), outputs the binary code of H

[0071] (6).

[0072] Represents the retrieval dataset. The binary code corresponding to each image , .

[0073] In order to prove the effectiveness of the proposed deep hash retrieval algorithm, this application first tests the mAP@all of four code lengths (16bit, 32bit, 48bit, 64bit) under a balanced data set. Figure 4 It can be observed that the deep hash retrieval algorithm based on meta-learning proposed in the present invention achieves the best retrieval effect in most cases. Although the training time of the three hash algorithms ADSH (Asymmetric Deep Supervised Hashing), FSSH (Fast Scalable Supervised Hashing) and CSQ (Central Similarity Quantization) is shorter than that of the algorithm proposed in the present invention, their retrieval results are far worse than those of the hash algorithm based on meta-learning. Therefore, at the cost of sacrificing a little extra time, the algorithm proposed in this application can better complete the retrieval tasks on the three benchmark datasets.

[0074] Since the deep hash retrieval algorithm based on meta-learning proposed in this application is mainly aimed at the imbalance problem in image retrieval, on three balanced datasets (CIFAR-10, STL-10 and CIFAR-100), the number of hash code bits is set to 32 bits, the number of data is set to five cases, and the imbalance ratios are 200, 100, 50, 20, and 10 respectively. The imbalance ratio refers to the ratio of the category with the largest amount of data to the category with the smallest amount of data. A comparison is made in the case of an imbalanced dataset, and the experimental results are as follows: Figure 5 As shown, in the CIFAR-10 and STL-10 datasets, the retrieval results of ADSH when the imbalance ratio is 10 are better than the other three methods. However, as the imbalance ratio increases, the experimental results clearly show that the retrieval results of the deep hash retrieval algorithm based on meta-learning proposed in the present invention are significantly better than the other three comparison methods. In the CIFAR-100 dataset, the retrieval results of ADSH, FSSH, CSQ and the algorithm proposed in the present invention are relatively low. The reason for this result may be that the number of CIFAR-100 classifications is relatively large, and the dataset consists of 100 categories. In this case, the experimental results of our algorithm are still better than other comparison methods, which can show that the deep hash retrieval algorithm based on meta-learning can effectively reduce the impact of imbalanced data in image retrieval.

[0075] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be considered exemplary and non-restrictive in all respects, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.

[0076] In addition, it should be understood that although the present specification is described in terms of implementation modes, not every implementation mode contains an independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A deep hash retrieval method based on meta-learning, characterized in that: The steps include: (1) Preprocess the image data and convert the balanced image dataset into an unbalanced image dataset; (2) Parameters of the meta-learning network , parameters of the image retrieval network Make updates; Using unbalanced image datasets in image retrieval networks Optimize by minimizing the loss of the unbalanced image dataset and reweighting the samples to obtain the optimal ; Using balanced image datasets in meta-learning networks Optimize, through the optimization of the image retrieval network To optimize , get the optimal ; Among them, the unbalanced image dataset is used in the image retrieval network. To optimize, define , Indicates samples, It is The label vector of the class, N is the total number of unbalanced image datasets, and the value of N is a natural number. represents the classifier, represents the classifier parameters; ①Use Calculate the minimization loss for the unbalanced image dataset and Recorded as , For the unbalanced image dataset Sample loss; ② When training unbalanced image datasets, use a weighted network Perform sample reweighting to improve training robustness, represents the meta-learning network parameters; ③Optimize by minimizing loss and reweighting samples Get the best ; （1）； (3) Construct a learning method for the image retrieval training process, divide the image retrieval network into multiple batches to iteratively train the images in the unbalanced image dataset, obtain the update equations of the image retrieval network parameters under the current number of iterative training times for multiple batches, and calculate the update of the meta-learning network parameters based on the feedback obtained from the updated parameters of the image retrieval network; (4) Update the parameters in the image retrieval training process. To improve the parameters during model training ; (5) Image retrieval generates hash codes. The input image is activated by the fully connected layer of the convolutional neural network (CNN) as a visual feature, and the visual feature is used for small-scale image retrieval.

2. The deep hash retrieval method based on meta-learning according to claim 1, characterized in that: The data of the same category form a category sample. The balanced image dataset is that the number of samples in each category is equal among samples of multiple categories; the unbalanced image dataset is that the number of samples in each category is different among samples of multiple categories.

3. The deep hash retrieval method based on meta-learning according to claim 1, characterized in that: Balancing image data represents the distribution of correctly labeled samples, Represents the first samples, Representative The label information of samples, and , is the number of balanced image dataset samples in the meta-learning network, obtained by formula (1) To optimize ; （2）， in , Calculated from metadata.

4. The deep hash retrieval method based on meta-learning according to claim 1, characterized in that: Construct a learning method for the image retrieval training process, divide the image retrieval network into multiple batches to iteratively train the images in the unbalanced image dataset, where each batch is ,in is the batch size, n <N; Then, each batch uses formula (1) Get the direction of the target loss L to move the current , t is the number of iterations, represents the number of updates, and the update equation of the image retrieval network parameters is obtained: （3）； Based on the feedback obtained from the updated parameters of the image retrieval network, the update of the meta-learning network parameters can be calculated: （4）； in, is the step size, m≤M.

5. The deep hash retrieval method based on meta-learning according to claim 1, characterized in that: Update the parameters in the image retrieval training process by updating the parameters To improve the parameters during model training ; （5）。

Images