Federal learning-based cross-modal encrypted data retrieval method and system, medium and processor
By adopting a cross-modal encrypted data retrieval method based on federated learning in cloud servers, the data leakage, low retrieval efficiency, insufficient personalized services and poor data privacy protection in encrypted data retrieval in cloud servers are solved, and efficient, secure and personalized data retrieval effects are achieved.
Patent Information
- Application Number
- CN202411512722.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2025-03-21
AI Technical Summary
When searching encrypted data in cloud servers, there are problems such as data leakage risk, low retrieval efficiency, insufficient personalized services and poor data privacy protection.
The cross-modal encrypted data retrieval method based on federated learning is adopted. Through the distributed federated learning architecture, the data is processed locally, and the encrypted search index and featured ciphertext are generated. The user queries in the encrypted state, and the cloud server performs ciphertext matching calculations, and returns the encrypted data results.
It effectively reduces the risk of data leakage, improves the security and privacy protection of the system, improves the speed and accuracy of data retrieval, enhances personalized services, and reduces system energy consumption and hardware costs.
Smart Images
Figure CN119670106A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of encrypted data retrieval, and in particular to a cross-modal encrypted data retrieval method, system, medium and processor based on federated learning. Background Art
[0002] With the rapid development of network and computing power, as well as the increasing popularity of social software and short video applications, massive multimodal data (including text, images, videos, and audio, etc.) is showing an explosive growth trend. These developments have put forward new challenges and demands for cross-modal data retrieval technology based on privacy protection and intelligent computing. First, the amount of multimodal data generated by the Internet is huge, which not only provides a rich source of fusion information, but also brings challenges in data storage, processing, and retrieval. Secondly, users in Internet applications expect to get more personalized and accurate retrieval results, which requires the system to be able to intelligently understand and analyze user preferences, historical behaviors, and contextual information. Finally, Internet applications, especially social media and short video applications, involve a large amount of personal data. How to perform effective data retrieval while protecting user privacy is an important issue that needs to be solved urgently. Therefore, our research focuses on improving the speed and accuracy of data retrieval, strengthening personalized services, strengthening data privacy protection, and using advanced artificial intelligence technologies to process and analyze large-scale multimodal data. These advances will further promote the integration and development of computer science, data science, artificial intelligence, and information security, and have important theoretical significance and application value.
[0003] With the gradual popularization of cloud service concepts and the continuous updating of functions, more and more enterprises and users store local data in cloud servers to realize data hosting and outsourcing. While reducing the burden of local storage management, users can still access data anytime and anywhere. In cloud storage and computing services, although it brings great convenience to users, it also faces many challenges, such as data security and privacy.
[0004] In order to prevent public cloud servers from obtaining sensitive information and protect data security and privacy, more and more users choose to encrypt their data before outsourcing it to cloud servers. Encryption can ensure the security of data because data cannot be read and used without the decryption key. However, for encrypted data stored in cloud servers, usability is also affected, for example, when users want to retrieve data of interest. The traditional solution is for users to download the data locally, decrypt it, and then retrieve it. However, this is unrealistic because it not only generates a lot of communication overhead, but also the storage and computing overhead caused by downloading data and then decrypting it is unacceptable to users. Therefore, in order to solve the problem of encrypted data retrieval on cloud servers, a searchable encryption technology is proposed, which allows users to store encrypted data on cloud servers and then retrieve it securely. Specifically, users first generate a search index for local data, and then encrypt it and upload it to the cloud server. When searching, users generate a search trap for the query content and submit it to the cloud server. Then, the cloud server matches the trap with the index in ciphertext format and returns the search results, i.e., encrypted data, to the user.
[0005] Traditional encrypted data retrieval technologies all use keywords to search for documents. However, there are also a large amount of multimedia data such as voice, images, audio and video on the Internet. Therefore, some researchers have proposed encrypted data retrieval based on multimedia data content. For example, in recent years, some people have proposed solutions for encrypted search of images or videos. Most of these solutions are based on tag and keyword searches, and cannot retrieve the content in the data. Therefore, some scholars have proposed encrypted data retrieval based on multimedia content. For example, some people have studied a method for secure retrieval of binary features of images, which extracts the local binary pattern value of the image and uses order-preserving encryption on it, realizing encrypted retrieval of images without reducing the search accuracy; then some scholars proposed encrypted data retrieval based on images, which uses deep learning to extract image features, and then uses a matrix with error learning to encrypt feature vectors, adding error values and random numbers to the matrix calculation to resist linear analysis attacks. However, cloud servers need to perform convolutional neural network calculations under ciphertext, which has high overhead. In practical applications, it may be necessary to search different types of data at the same time, such as using text queries to obtain related images or videos. Therefore, some researchers have proposed cross-modal search, that is, using the semantics of one modality to query data of other modalities. Traditional cross-modal search mainly uses statistical analysis methods, such as canonical correlation analysis and cross-modal factor analysis. However, traditional analysis methods are difficult to model cross-modal data in actual application scenarios. Subsequently, researchers proposed to use cross-modal hashing to retrieve data, which maps data of different modalities into a common Hamming space to achieve cross-modal search. However, the search accuracy of cross-modal hashing retrieval is not high and the effect is unsatisfactory. In order to improve the search accuracy, researchers proposed cross-modal search based on intelligent computing. For example, some scholars proposed a search method based on deep learning. They use a deep network with a highly nonlinear structure, which is more suitable for associating and mining intra-modal features and inter-modal semantic relationships. However, deep learning requires a large amount of high-quality cross-modal data to obtain accurate search results. In actual deployment, cross-modal data may be stored in each user, and the central server may have little or no data. Due to privacy protection and broadband speed, these data are difficult to aggregate, so the search accuracy often does not meet the expected effect.
[0006] Comparing the research work on data retrieval of different modalities at home and abroad, we can find the following shortcomings:
[0007] (1) Traditional data retrieval technologies are mostly based on keywords or text, which is inadequate in the face of multi-format big data models.
[0008] (2) Traditional cross-modal search uses statistical analysis methods, which is not effective for cross-modal data modeling in practical application scenarios.
[0009] (3) Currently, most cross-modal data retrieval methods based on intelligent computing are performed in a plaintext environment, which may pose a threat to users' personal privacy and data security. Summary of the invention
[0010] In view of the problems existing in the prior art, the present invention provides a cross-modal encrypted data retrieval method based on federated learning, which can retain data locally through a distributed federated learning architecture, reduce the risk of data leakage, and greatly improve the security and privacy protection of the system. The specific technical solution is as follows:
[0011] A cross-modal encrypted data retrieval method based on federated learning, comprising:
[0012] Initialize the algorithm, the collaborative server generates the public and private key pair of the Paillier system, and each data owner randomly generates a key;
[0013] Use federated learning to obtain a global search model;
[0014] The data owner generates a search index using a global search model, wherein the search index includes an index identifier and a feature ciphertext, and sends the index to a cloud server;
[0015] The user sends query permissions to the data owner and generates a query trap and sends it to the cloud server;
[0016] The cloud server parses the query trap to obtain the aggregate bucket value, calculates the threshold, encrypts it, and sends it to the collaborative server. The collaborative server decrypts it and obtains the corresponding data to form a set. After obtaining the set, the cloud server arranges each piece of data in ascending order and sends it to the user.
[0017] Preferably, the global search model is obtained by adopting federated learning, specifically as follows:
[0018] Local training: Each data owner uses local data to train a local search model and calculates the training weight based on the number of data samples. The data owner multiplies the local search model by the training weight and encrypts it using the public key of the collaborative server and uploads it to the cloud server.
[0019] Model aggregation, after receiving the encrypted data uploaded by the participating data owners, the cloud server performs ciphertext multiplication to obtain the encrypted global search model and sends it to the collaborative server;
[0020] Local update: The collaborative server decrypts the global search model to obtain Wt+1 and sends it to all data owners. Each data owner sets the value of the local search model W1 to Wi+1.
[0021] Preferably, the data owner generates a search index using a global search model, the search index includes an index identifier and a feature ciphertext, and sends it to a cloud server, as follows:
[0022] Feature extraction, after obtaining the global search model using federated learning, the data owner uses the model to extract a unified feature vector from the local data;
[0023] Generate an index identifier. The data owner generates an index identifier based on the feature vector set;
[0024] To generate feature ciphertext, the data owner first expands the extracted data features. The data owner encrypts the expanded vector using the kNN algorithm. The data owner sends its index tag and feature ciphertext to the cloud service.
[0025] Preferably, the user sends a query permission to the data owner, and generates a query trap and sends it to the cloud server, as follows:
[0026] When the data owner authorizes the user to query permissions, the authorization key is sent to the user, and then the user splits the extended vector into two. The user uses the matrix key to encrypt the two split vectors, and finally the user sends the query trap to the cloud server.
[0027] Preferably, when the aggregate bucket value is sent to the collaboration server for decryption, the cloud server generates a random number, encrypts it, and multiplies it by the aggregate bucket value.
[0028] A cross-modal encrypted data retrieval system based on federated learning, applied to the method described, comprising a user module, a data owner module, a federated learning module, a collaborative server module and a cloud server module;
[0029] The user module is used to send a query instruction and generate a query trapdoor;
[0030] The data owner module is used to generate a search index and provide a data source for retrieval by the user module;
[0031] The federated learning module is used to generate a global search model;
[0032] The collaboration server module is used to generate a public and private key pair of the Paillier system;
[0033] The cloud server module is used to parse the query trap and feed back the queried data set to the user.
[0034] A computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method described.
[0035] A processor is used to run a program, wherein the method described is executed when the program is run.
[0036] Compared with the prior art, the present invention has the following beneficial effects:
[0037] In the present invention, firstly, through the distributed federated learning architecture, the data can be retained locally, reducing the risk of data leakage and greatly improving the security and privacy protection of the system. At the same time, the distributed nature of the method enhances the stability of the system, and even if some nodes fail, it will not affect the overall function. In addition, users can directly retrieve data in an encrypted state, which simplifies the operation process and improves the convenience of use and work efficiency. The local training method of federated learning also effectively reduces the demand for data transmission, reduces system energy consumption, and thus reduces energy consumption and carbon footprint. Finally, since there is no need to build a complex centralized server, this method significantly reduces the hardware and maintenance costs of the system, making the overall solution more economical and efficient. Through the above advantages, this method not only ensures the security and privacy of the data, but also realizes efficient, stable and energy-saving cross-modal data retrieval, and has broad application prospects and application value. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.
[0039] Figure 1 is a flow chart of the method of the present invention;
[0040] Figure 2 Take a screenshot of the initialization algorithm code;
[0041] Figure 3 Generate algorithm code screenshots for federated learning models;
[0042] Figure 4 Screenshot of the algorithm code for generating index identification;
[0043] Figure 5 Screenshot of the algorithm code for generating feature ciphertext;
[0044] Figure 6 Screenshot of the code for generating the trapdoor algorithm;
[0045] Figure 7 Flowchart for search algorithm code;
[0046] Figure 8 This is the system model diagram. DETAILED DESCRIPTION
[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0048] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0049] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0050] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
[0051] Example
[0052] like Figure 1-8 As shown in the figure, a cross-modal encrypted data retrieval method based on federated learning is defined as follows for the convenience of explanation: let F be a permutation function with a key, (H1, H2, H3, ..., H m ) are m local sensitive hash functions. Let KeyGen be the algorithm for generating the public and private key pairs of the Paillier system, Enc and Dec be the encryption and decryption algorithms of the Paillier system
[0053] The specific steps include:
[0054] S1. Initialize the algorithm. The collaborative server generates the public and private key pair of the Paillier system, and each data owner randomly generates a key. The details are as follows:
[0055] Initialization algorithm as algorithm as Figure 2As shown in Algorithm 6-1, first, the collaborative server generates the public-private key pair of the Paillier system (line 4); secondly, each data owner randomly generates a key (k, S, M1, M2), where k is the permutation function key, S is a d+1-dimensional binary vector, and (M1, M2) are two (d+1)×(d+1) reversible matrices (lines 7-9).
[0056] S2. Use federated learning to obtain a global search model; the details are as follows:
[0057] (1) Model generation algorithm: In the federated learning model generation algorithm, the data owner, cloud server, and collaborative server jointly train the global search model. The learning process is as follows: Figure 3 As shown in Algorithm 6-2. The algorithm performs T rounds of training, each round of training is divided into three steps, and randomly selects some data owners to participate (line 4). The specific steps are as follows:
[0058] Step 1: Local training. Each data owner uses local data to train a local search model (line 6), and calculates the training weight according to the number of data samples (in this embodiment, the training weight is set to the number of local samples divided by the number of global samples) (line 7). Then, the data owner multiplies the local search model by the training weight and encrypts it using the public key of the collaborative server and uploads it to the cloud server (line 9).
[0059] Step 2: Model aggregation, after receiving the encrypted data uploaded by the participating data owners, the cloud server t-1 , perform ciphertext multiplication to obtain the encrypted global search model CW t+1 , and sent to the collaboration server.
[0060] Step 3: Local update, the collaborative server decrypts the global search model to obtain W t+1 , and sent to all data owners (line 15), each data owner sets the value of the local search model W1 to W i+1 .
[0061] S3. The data owner generates a search index using a global search model. The search index includes an index identifier and a feature ciphertext, and sends the index to the cloud server. The specific steps are as follows:
[0062] Step 1: Feature Extraction,After using federated learning to obtain the global search model, the data owner uses this model to extract a unified feature vector from the local data. For example, data owner i, whose data set is O i , then the extracted feature vector set is P i .
[0063] Step 2: Generate index identifier. The data owner generates index identifier based on the feature vector set. The calculation process is as follows: Figure 4 As shown in Algorithm 6-3, the data owner uses m LSH functions to calculate the feature vector. The value range of the LSH function is l, so the data owner will get 1 bucket value, each value is an n-bit vector.
[0064] The specific process is as follows: the data owner first initializes the vector value of a bucket to 0 (lines 4-5), and uses the LSH function to calculate all feature vectors. For example, if feature P is mapped to m buckets, the jth bit of the vector value of these m buckets is set to 1 (lines 6-9). After calculating all feature vectors, the data owner permutes the bucket number (line 11), and encrypts the bucket value using the public key of the collaborative server (line 12), and finally obtains the encrypted index identifier {c}.
[0065] Step 3: Generate feature ciphertext, such as Figure 5 As shown in Algorithm 6-4, the data owner first expands the extracted data features to d+1 dimensions (line 6); secondly, the data owner uses the kNN algorithm to encrypt the expanded vector: first, use S to split the expanded vector into two, and then use S i For each bit, when S = 0, let the values of p and j equal to p, and when S = 1, let the sum of p and p equal to p; then, the data owner uses the matrix key to encrypt the two split vectors (lines 7-13). Finally, the data owner sends its index tag {c]} and feature ciphertext {c(P)} to the cloud server.
[0066] S4. The user sends the query permission to the data owner and generates a query trap and sends it to the cloud server. The specific steps are as follows:
[0067] Specific as Figure 6 As shown in Algorithm 6-5, when the data owner authorizes the user to query the data, the authorization key (k, S, M1, M2) is sent to the user, and the user obtains it (line 9). Then, the user uses S i The extended vector is split into two, and the user uses the matrix key to encrypt the two split vectors to generate TD2 (lines 10-16). Finally, the user sends the query trapdoor (TD1, TD2) to the cloud server (line 17).
[0068] S5. The cloud server parses the query trap to obtain the aggregate bucket value, calculates the threshold, encrypts it, and sends it to the collaboration server. The collaboration server decrypts it to obtain the corresponding data and forms a set. After obtaining the set, the cloud server sorts each piece of data in ascending order and sends it to the user, as follows:
[0069] like Figure 7As shown in Algorithm 6-6 in , when receiving the user's query trap, the cloud server first parses TD1 to obtain the bucket number, and performs ciphertext multiplication on the encrypted value of the corresponding bucket to obtain the aggregate bucket value c (lines 4-5); then, the aggregate bucket value needs to be sent to the collaborative server for decryption. In order to prevent it from obtaining the real aggregate bucket value, the cloud server generates a random number, encrypts it and multiplies it by the aggregate bucket value. Secondly, the cloud server calculates the threshold cthold and encrypts it. The threshold is used to filter candidate data, and its value is set to the sum of m and r (line 8), where m is the number of LSH functions. Finally, the cloud server sends the encrypted aggregate bucket value and threshold (c, cthold) to the collaborative server (line 9). The collaborative server decrypts to obtain the aggregate bucket value and threshold (c, cthold), and then the collaborative server determines each bit value of 0. If it is greater than or equal to the threshold, the data corresponding to the bit will be added to the set S and returned to the cloud server (lines 11-13). After obtaining the set S, the cloud server calculates the score for each piece of data in S (line 16) and arranges them in ascending order. Finally, the cloud sends the first k pieces of data in the permutation to the user (line 18).
[0070] Example 2
[0071] A cross-modal encrypted data retrieval system based on federated learning, applied to the method described, comprising a user module, a data owner module, a federated learning module, a collaborative server module and a cloud server module;
[0072] The user module is used to send a query instruction and generate a query trapdoor;
[0073] The data owner module is used to generate a search index and provide a data source for retrieval by the user module;
[0074] The federated learning module is used to generate a global search model;
[0075] The collaboration server module is used to generate a public and private key pair of the Paillier system;
[0076] The cloud server module is used to parse the query trap and feed back the queried data set to the user.
[0077] Example 3
[0078] A computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method described.
[0079] Example 4
[0080] A processor is used to run a program, wherein the method described is executed when the program is run.
[0081] Those of ordinary skill in the art will appreciate that the units of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0082] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored, etc.
[0083] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0084] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nlyMemory), random access memory (RAM, RandomAccessMemory), mobile hard disk, magnetic disk or optical disk, etc., which can store program code.
[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.
Claims
1. A cross-modal encrypted data retrieval method based on federated learning, characterized in that: include: Initialize the algorithm, the collaborative server generates the public and private key pair of the Paillier system, and each data owner randomly generates a key; Use federated learning to obtain a global search model; The data owner generates a search index using a global search model, wherein the search index includes an index identifier and a feature ciphertext, and sends the index to a cloud server; The user sends query permissions to the data owner and generates a query trap and sends it to the cloud server; The cloud server parses the query trap to obtain the aggregate bucket value, calculates the threshold, encrypts it, and sends it to the collaborative server. The collaborative server decrypts it and obtains the corresponding data to form a set. After obtaining the set, the cloud server arranges each piece of data in ascending order and sends it to the user.
2. According to the cross-modal encrypted data retrieval method based on federated learning according to claim 1, it is characterized in that: The global search model is obtained by adopting federated learning, as follows: Local training: Each data owner uses local data to train a local search model and calculates the training weight based on the number of data samples. The data owner multiplies the local search model by the training weight and encrypts it using the public key of the collaborative server and uploads it to the cloud server. Model aggregation, after receiving the encrypted data uploaded by the participating data owners, the cloud server performs ciphertext multiplication to obtain the encrypted global search model and sends it to the collaborative server; Local update: The collaborative server decrypts the global search model to obtain Wt+1 and sends it to all data owners. Each data owner sets the value of the local search model W1 to Wi+1.
3. According to a cross-modal encrypted data retrieval method based on federated learning according to claim 1, it is characterized in that: The data owner generates a search index using a global search model, the search index including an index identifier and a feature ciphertext, and sends it to the cloud server, as follows: Feature extraction, after obtaining the global search model using federated learning, the data owner uses the model to extract a unified feature vector from the local data; Generate an index identifier. The data owner generates an index identifier based on the feature vector set; To generate feature ciphertext, the data owner first expands the extracted data features. The data owner encrypts the expanded vector using the kNN algorithm. The data owner sends its index tag and feature ciphertext to the cloud service.
4. According to a cross-modal encrypted data retrieval method based on federated learning according to claim 1, it is characterized in that: The user sends query permissions to the data owner and generates a query trap and sends it to the cloud server, as follows: When the data owner authorizes the user to query the permission, the authorization key is sent to the user, and then the user splits the extended vector into two. The user uses the matrix key to encrypt the two split vectors, and finally the user sends the query trap to the cloud server.
5. According to a cross-modal encrypted data retrieval method based on federated learning according to claim 1, it is characterized in that: When the aggregate bucket value is sent to the collaboration server for decryption, the cloud server generates a random number, encrypts it and multiplies it by the aggregate bucket value.
6. A cross-modal encrypted data retrieval system based on federated learning, characterized in that: The method applied to any one of claims 1 to 5, comprising a user module, a data owner module, a federated learning module, a collaborative server module, and a cloud server module; The user module is used to send a query instruction and generate a query trapdoor; The data owner module is used to generate a search index and provide a data source for retrieval by the user module; The federated learning module is used to generate a global search model; The collaboration server module is used to generate a public and private key pair of the Paillier system; The cloud server module is used to parse the query trap and feed back the queried data set to the user.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 5.
8. A processor, characterized in that: The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 5 when running.