A federated image retrieval method based on a hash model

Abstract

The application provides a federated image retrieval method based on a hash model, comprising the following steps: a server end sends an initial hash model to each selected client end; a client end reads training data from a local database to update the model, and sends the updated hash model back to the server end; the server end aggregates the hash models sent back by the client ends to obtain a new round of global hash model; the server end sends the global hash model of the current round to each selected client end; the client end makes the received global model and the local model of the last round learn from each other according to the set batch size and the local training round number, and sends the updated global model to the server end; the iteration is repeated until a termination condition is reached, i.e., the iteration is terminated; each client end performs image retrieval according to the trained local model to obtain a retrieval result. The application applies the horizontal federated learning technology to image retrieval in combination with image hash, and can quickly improve the image retrieval accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing and federated learning, and in particular, especially relates to a federated image retrieval method based on a hash model. BACKGROUND

[0002] In the pedestrian re-identification task, the photographed pedestrian images in the monitoring camera are retrieved, but the monitoring video has strong privacy, and the monitoring videos in different regions cannot be transmitted at will, that is, there is a "data island" problem.

[0003] In the face of the crazy growth of multimedia data in the big data era, the image hash algorithm uses binary code to describe the image, and has relatively high calculation speed and low storage cost. The traditional image hash algorithm obtains the hash code by binarizing the manually extracted features. In recent years, in order to extract more rich information from images, various hash algorithms based on deep learning have been proposed, which use convolutional neural networks to extract image features and then binarize to obtain hash codes. According to the different supervision information, the hash algorithm can be divided into supervised hash and unsupervised hash.

[0004] Supervised hash uses the label information of training data to obtain better performance, and the design of the method mainly includes the design of the network and the design of the loss function two aspects, including DSH, CSQ and other algorithms. Unsupervised hash does not use any label information, so it is more difficult to improve the performance compared with supervised hash. It obtains similarity information through the distance between features, which can be divided into three categories: dissimilarity deep hash, deep hash based on generative model and deep hash based on pseudo-label learning to transform unsupervised problem into supervised problem, including DeepBit, HashGAN and other algorithms. Most deep hash models are trained on large-scale data sets, and all data should be concentrated in one place. However, data is usually stored in scattered places or organizations (such as on smart phones or in institutions), and collecting these data will at least encounter two problems. On the one hand, the more valuable the data (such as monitoring videos, financial records, medical treatment records, etc.), the more sensitive it is, and the institution will not agree to upload the data, that is, the data island problem. On the other hand, there is a risk of privacy leakage in the process of collecting such data, which causes the problem of privacy leakage.

[0005] The proposal of federated learning provides a possibility for the solution to this problem. This new deep learning framework aims to achieve data privacy protection through a model sharing instead of data sharing. Through federated learning, we do not need to collect data, but make multiple participants who have data jointly train the model, and the central server coordinates each participant to get the final global model.

[0006] According to different user IDs and data characteristics, federated learning can be divided into horizontal federated learning, vertical federated learning and federated transfer learning. Among them, horizontal federated learning refers to the case that the data characteristics of participants overlap more, and the user IDs overlap less, such as two banks serving different regions, the users overlap less but the data characteristics may be similar, and a global model can be trained together, which is consistent with our motivation. In horizontal federated learning, multiple participants with the same data structure cooperatively train the model through parameters or cloud servers, the server distributes the initial model to each participant, each party independently trains the model using local data, uploads the updated model to the central server after training is completed, aggregates the global model, and the server sends the global model back to the local, and the local starts a new round of training, and iterates multiple times to complete the training.

[0007] The most widely used aggregation method in the federated learning algorithm at present is FedAvg, which performs weighted average on the model parameters from each participant to obtain a global model. However, in reality, the data from different clients is often heterogeneous, that is, the data distribution contained by different clients is unbalanced, which will reduce the performance of the global model. The reason is that the difference in data distribution makes the local model develop in different directions, causing the deviation of the global model. Applying the deviated model to image retrieval will inevitably reduce the retrieval performance. SUMMARY

[0008] In view of the deficiencies of the prior art, the present application provides a federated image retrieval method based on a hash model. The present application applies horizontal federated learning technology combined with image hash to the field of image retrieval, mainly focusing on supervised image hash, realizes a privacy-protected image retrieval method, and introduces mutual learning technology to deal with the data heterogeneity problem of participants.

[0009] The technical means adopted by the present application are as follows:

[0010] A federated image retrieval method based on a hash model is applied to a distributed network system, the distributed network system comprising a server end and M clients, the clients having data and labels, the server end having no data and labels, the clients mainly using local data to train a hash model, the server end mainly initializing, distributing and aggregating the hash model, and coordinating the clients to obtain a trained hash model;

[0011] The method comprises the following steps:

[0012] S1, after the server end initializes the hash model parameters, randomly selects the clients participating in the first round of training according to a preset proportion, and sends the initial hash model to each selected client;

[0013] S2, each selected client receives an initial global hash model, reads training data from a local database, updates the model according to a set batch size and a local training round number, and sends the updated hash model back to the server end while saving the local model;

[0014] S3, the server end aggregates the hash models sent back by the clients to obtain a new round of global hash model;

[0015] S4, the server end selects clients participating in the current round of training according to a preset proportion and sends the current round of global hash model to each selected client;

[0016] S5, each selected client receives the current round of hash model and makes the received global model and the local model of the last round learn from each other according to a set batch size and a local training round number, sends the updated global model to the server end, and keeps the updated local model;

[0017] S6, repeating S3-S5 until a termination condition, i.e., iteration termination, is reached;

[0018] S7, each client performs image retrieval according to the trained local model and obtains a retrieval result.

[0019] Further, the hash model is a deep hash model, which includes a feature extraction layer and a hash layer;

[0020] The feature extraction layer is a convolutional neural network;

[0021] The hash layer is a fully connected layer, the input size of which is the output size of the convolutional neural network, and the output size is a preset hash bit number.

[0022] Further, the parameter update algorithm of the hash model is one of DSH, GreedyHash and CSQ algorithms.

[0023] Further, the server end aggregates the hash models sent back by the clients to obtain a new round of global hash model, which includes an average aggregation or a weighted average aggregation.

[0024] Further, the termination condition is that the number of iterations reaches a set global round number.

[0025] Further, the client makes the received global model and the local model of the last round learn from each other according to a set batch size and a local training round number, which includes adding KL divergence to the original hash loss function to make the local model saved in the last round and the global model knowledge fusion.

[0026] Compared with the prior art, the present application has the following advantages:

[0027] The application intercepts pedestrian images of a monitoring video to form a data set, different regions use local data to train a hash model, and then complete collaborative training of the model under the method to obtain a high-performance retrieval model suitable for the local. When an image of a query person is input, the accuracy of the searched pedestrian image is higher. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0029] Figure 1 The federal model training flowchart of the present application.

[0030] Figure 2 The hash model parameter updating flowchart of the present application. DETAILED DESCRIPTION

[0031] In order to make the person skilled in the art better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely in the following by combining the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0032] Image hashing is an algorithm that converts images into binary codes, which is widely used in large-scale image retrieval due to its simple calculation and convenient storage. Image retrieval refers to finding similar images in semantic space or feature space when an image is input. The most direct method is to use a feature to represent an image and find the most similar image by calculating the Euclidean distance between the query image and the database image in the feature space. However, image hashing converts images into binary codes, which greatly reduces the amount of calculation and storage space. Based on the above purpose, similar images in image hashing are expected to get similar hash codes, and dissimilar images are expected to get significantly different hash codes. Current hashing algorithms are based on centralized data sets for training, which is difficult to implement for sensitive data such as medical images, and also poses a risk of privacy leakage during data transmission. Federated learning is a distributed machine learning framework that aims to break down data silos and protect data privacy by replacing data transmission with model transmission. Horizontal federated learning refers to a pattern where data features overlap more and sample IDs overlap less, which is suitable for multiple institutions in the same industry to jointly train a model. Due to the differences in data labels among multiple institutions in real-world scenarios, there is a common data heterogeneity problem in horizontal federated learning, which causes the model to deviate from the local data and even perform worse than the model trained only with local data.

[0033] Based on the above background, the present application discloses a federated image retrieval method based on a hash model, which is applied to a distributed network system. The distributed network system includes one server end and M client ends. The client ends have data and labels, and the server end has no data and labels. To solve the common data heterogeneity problem in horizontal federated learning, the present application introduces mutual learning technology. The method principle diagram is shown in the accompanying Figure 1 The left part is the entire horizontal federated hashing framework, and the right half is the detailed process of the client training, which includes a knowledge fusion module. Two models perform knowledge fusion through mutual learning, so that the final model has both global knowledge and local knowledge, thereby obtaining a personalized model suitable for the local.

[0034] Specifically, it is assumed that the jth client has N images, where j = 1, 2, 3,..., M. The local data is represented as The corresponding label is Therefore, the dataset composed of the client ends is X = {X 1 , X 2 ,..., X M}. In addition, the proportion of the client end participating in training each round is F, the batch size during local training is B, the number of local training rounds is E, and the number of global training rounds is T.

[0035] The method steps are shown in the accompanying Figure 2As shown, specifically as follows:

[0036] S1, the server end initializes the hash model parameter ω1, randomly selects the client set M1 according to the set first round of the client participation training ratio F1, and sends the initial hash model to M1;

[0037] S2, the client j∈M1 receives the hash model ω1, reads the training data from the local database , updates the model according to the set batch size B and the local training round E, and sends the updated hash model back to the server end while saving a copy of the model On the local, the client performs the process in parallel;

[0038] Further, the model update algorithm can select any depth hash algorithm, including DSH, CSQ, etc., and the loss function of the model update is L h , and if the CSQ algorithm is selected, the loss function is L CSQ .

[0039] Specifically, the depth hash refers to an algorithm for converting an image into binary code by using a deep neural network, and the network is composed of a feature extraction layer and a hash layer. The feature extraction layer is generally a convolutional neural network, and the hash layer is a fully connected layer with an output size of the hash bit number, i.e. the number of bits of the binary code. Given a training set containing N images, the depth hash model learns a nonlinear mapping function F: x→b∈{-1,+1} k , where k is the hash code length. Its loss function can be expressed as:

[0040]

[0041] Where S represents the feature space, i.e. minimizing the distance between the hash code and the feature space to keep the similarity between the hash space and the feature space consistent.

[0042] The hash algorithm includes supervised algorithm and unsupervised algorithm, and the difference is mainly in the loss function construction. For the supervised algorithm, it can be roughly divided into pair algorithm, ternary algorithm and center similarity algorithm. In the pair algorithm, the input is two pictures, and the similarity of similar pictures is 1 and the similarity of dissimilar pictures is 0. In the ternary algorithm, the input is three pictures, including anchor, positive sample and negative sample, and the similarity between the anchor and the positive sample is greater than the similarity between the anchor and the negative sample. The center similarity algorithm uses class label as similarity information, so that the images belonging to a class are close to the class center in the hash space. The classic hash algorithm includes DSH, GreedyHash, CSQ, etc., and the CSQ algorithm preferred by the embodiment belongs to the center similarity algorithm, and its loss function can be expressed as:

[0043]

[0044] Where c represents the category, c = 1, 2, ..., C. s represents the similarity score; if image i belongs to category c, s... i，c The value is 1 if the expression is true and 0 otherwise. h is the approximate hash code. The second term of the loss function is a penalty term used for discrete constraints on the hash code.

[0045] S3, The hash model sent back by the server to the client. Aggregation is performed to obtain a new global model ω. t ;

[0046] Specifically, various aggregation algorithms can be selected, including average aggregation, weighted average aggregation, etc. In this embodiment, the weighted average aggregation algorithm is preferred, and the formula can be expressed as follows:

[0047]

[0048] S4. The server then randomly selects client Mt again based on the client participation ratio Ft set in this round, and sends the new global model ω to the server. t Send to Mt;

[0049] S5. The client j∈Mt, according to the set batch size B and local training rounds E, makes the received global model ω t Compared to the previous round of local models They learn from each other and then update the global model. Send to the server to retain the updated local model. The client-side execution of this process is parallel.

[0050] Specifically, mutual learning is achieved through KL divergence. For the global model, its KL divergence can be expressed by the following formula:

[0051]

[0052] Similarly, for the local model, its KL divergence can be expressed as:

[0053]

[0054] Where p is the classification prediction probability of the image, and z is the output prediction of the image after passing through the fully connected layer and before entering the softmax layer. The subscript g indicates the global model, and the subscript l indicates the local model. Therefore, For image x i The predicted probability of belonging to class c after passing through the local model classifier. For image x ithe prediction probability of the image x the prediction of the feature outputted by the local model for the image x the prediction of the feature outputted by the global model for the image x

[0055] Since the calculation of KL divergence needs the output of the classifier, the present application adds a fully connected layer after the feature output layer of the original hash model, as shown in the following figure. Figure 1 The output size of the fully connected layer is the number of categories, which can also be called a classifier.

[0056] To make the model more robust, cross-entropy loss is added as one of the loss functions when mutual learning is performed, and the formula is as follows:

[0057]

[0058] where the I(·) function represents the indicator function, that is, 1 when the condition is met, and 0 when the condition is not met. When the label y i of the image x i is the same as the category c, the prediction probability of the category c is calculated as

[0059] Specifically, the loss function of the updated global model is composed of three parts, and is expressed as:

[0060] L g = L h-g + αL kl-g + βL c-g

[0061] The loss function of the local model is expressed as:

[0062] L l = L h-l + αL kl-l + βL c-l

[0063] where α and β are hyperparameters set to control the degree of mutual learning.

[0064] S6, repeat S3-S5 until the iteration reaches the set global round T, that is, the iteration is terminated.

[0065] Finally, the client obtains the personalized model belonging to the local

[0066] S7, each client performs image retrieval according to the trained local model to obtain the retrieval result.

[0067] In the pedestrian re-identification task, the photographed pedestrian image in the monitoring camera can be searched using a hash algorithm. However, the monitoring video has strong privacy, and the monitoring videos in different regions cannot be transmitted at will, so the application breaks the 'data island' in the monitoring video. The pedestrian image of the monitoring video is intercepted to form a data set, different regions use local data to train a hash model, and then the model is collaboratively trained under the method to obtain a high-performance retrieval model suitable for the local. When the image of the query person is input, the accuracy of the searched pedestrian image can be higher.

[0068] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A federated image retrieval method based on a hash model, applied to a distributed network system, the distributed network system comprising one server end and M client ends, the client ends having data and labels, and the server end having no data and labels, the client ends mainly using local data to train the hash model, and the server end mainly initializing, distributing and aggregating the hash model to coordinate the client ends to obtain a trained hash model; the method comprising the following steps: characterized in that S1, after the server end initializes hash model parameters, randomly selecting client ends participating in the first round of training according to a preset proportion, and sending the initial hash model to each selected client end; S2, each selected client end receives the initial global hash model, reads training data from a local database, updates the model according to a set batch size and local training rounds, sends the updated hash model back to the server end, and saves the local model; S3, the server end aggregates the hash models sent back by the client ends to obtain a new round of global hash model; S4, the server end again selects client ends participating in the current round of training according to a preset proportion, and sends the global hash model of the current round to each selected client end; S5, each selected client end receives the hash model of the current round, and makes the received global model and the local model of the last round learn from each other according to a set batch size and local training rounds, sends the updated global model to the server end, and retains the updated local model; S6, repeating S3-S5 until a termination condition, i.e., iteration termination, is reached; S7, each client end performs image retrieval according to the trained local model to obtain a retrieval result. The hash model is a deep hash model, comprising a feature extraction layer and a hash layer; 2.The federated image retrieval method based on hash model according to claim 1, wherein, The feature extraction layer is a convolutional neural network; The hash layer is a fully connected layer, the input size of which is the output size of the convolutional neural network, and the output size of which is a preset hash bit number. The parameter update algorithm of the hash model is one of DSH, GreedyHash and CSQ algorithms. 3.The federated image retrieval method based on hash model according to claim 2, characterized in that, The server end aggregates the hash models sent back by the client ends to obtain a new round of global hash model, including by means of average aggregation or weighted average aggregation.

4. The federated image retrieval method based on hash model according to claim 1, characterized in that, The termination condition is that the number of iterations reaches a set global round number.

5. The federated image retrieval method based on hash model according to claim 1, characterized in that, The client end makes the received global model and the local model of the last round learn from each other according to a set batch size and local training rounds, including adding KL divergence to the original hash loss function to make the local model saved in the last round and the global model knowledge fusion.

6. The federated image retrieval method based on hash model according to claim 1, characterized in that, ​

Images