A homomorphic encryption method for a deep learning model of natural language processing

Abstract

The application discloses a homomorphic encryption method for a deep learning model for natural language processing, and first obtains an original symbol table according to data to be encrypted or a deep learning model to be used; generates a random symbol table which does not contain repeated symbols and is not less than the original symbol table in size according to the original symbol table, and the symbols in the original symbol table and the symbols in the random symbol table form a one-to-one or one-to-many mapping relationship; the random symbol table is used as a key K; a piece of plaintext data corresponding to the deep learning model is divided into symbol strings through a word segmentation algorithm; the symbols in the symbol strings are mapped into new symbols according to the key K, and are concatenated to generate ciphertext m encrypted by the key K. The ciphertext generated by the method of the application can keep the calculation homomorphism in deep learning, that is, the deep learning model can be trained and predicted based on the ciphertext, the calculation efficiency of the training is not affected, the prediction result is not changed, and the accuracy and recall rate effects are not changed.

Description

Technical Field

[0001] This invention relates to the fields of computer data security and cryptography, and in particular to a homomorphic encryption method for deep learning models used in natural language processing. Background Technology

[0002] Services based on deep learning-based natural language processing are becoming increasingly common, with natural language and big data-driven recommendation and recognition services bringing greater convenience to users. However, the process of users submitting data may lead them to sacrifice their privacy. To ensure service providers can continue offering services while protecting user privacy, several technologies are under research in the field of privacy computing. It's important to note that privacy computing is not a single technology; rather, it's an interdisciplinary field integrating cryptography, artificial intelligence, computer science, and even security hardware. Currently, widely researched technologies include: secure multi-party computation, federated learning and trusted execution environments, differential privacy, and homomorphic encryption.

[0003] The first category comprises purely cryptographic technologies, such as Secure Multi-Party Computation (SMPC or MPC); the second category includes technologies that integrate artificial intelligence and cryptography, such as Federated Learning; and the third category consists of technologies based on trusted hardware, such as Trusted Execution Environments (TEEs). In addition to these three main technological directions, technologies such as homomorphic encryption and differential privacy are also widely adopted, either integrated into the above-mentioned directions as part of the solution or used independently. It is worth noting that these technologies are not mutually exclusive; they are often used as needed, or even in combination, depending on the specific scenario, to achieve business goals. A comparison of privacy computing technologies is as follows.

[0004] Multi-party secure computation and fully homomorphic encryption algorithms offer high security, but require significant engineering modifications and cause a substantial decrease in computational and storage performance, making them difficult to implement in practice. Trusted execution environments typically require specialized hardware systems; while offering superior security and computational performance, their application scenarios are severely limited. Differential privacy and federated learning technologies represent a compromise between security and computational performance, but overall, computational speed and accuracy still suffer to some extent.

[0005] In general, while some of these algorithms offer high security, they cause massive storage expansion and a significant drop in computational performance. Others, while still competitive in terms of computing speed, require substantial engineering modifications, involving a complete overhaul of the transmission and deep learning computation processes, which in turn reduces the accuracy of deep learning. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a homomorphic encryption method for deep learning models used in natural language processing. This encryption method has superior computational performance, enabling millisecond-level encryption and decryption without affecting the speed of deep learning models.

[0007] A homomorphic encryption method for deep learning models in natural language processing includes the following steps:

[0008] Step 1: Key generation;

[0009] Obtain an original symbol table based on the data to be encrypted or the deep learning model to be used; generate a random symbol table without duplicate symbols based on the original symbol table, wherein the size of the random symbol table is not smaller than the original symbol table, and the symbols in the original symbol table and the symbols in the random symbol table form a one-to-one or one-to-many mapping relationship; the random symbol table serves as the key K.

[0010] Step Two: Data Encryption;

[0011] Step (2.1): Divide plaintext data into symbol strings using a word segmentation algorithm;

[0012] Step (2.2): Map the symbols in the symbol string from the original symbols to new symbols according to the key K; wherein, if the mapping relationship is a one-to-one mapping, it is directly mapped to the new symbol; if the mapping relationship is a one-to-many mapping, a symbol is randomly selected from the multiple symbols mapped from the original symbol as the new symbol.

[0013] Step (2.3): Concatenate the new symbols to generate ciphertext m encrypted with key K.

[0014] Further, a random symbol table without duplicate symbols is generated based on the original symbol table, specifically including:

[0015] The original symbol table is randomly shuffled to generate a random symbol table; at this time, the symbols in the original symbol table and the symbols in the random symbol table form a one-to-one mapping relationship.

[0016] Alternatively, select a new symbol table with a size no smaller than the original symbol table, randomly shuffle the order, and generate a random symbol table.

[0017] Furthermore, the original symbol table is randomly shuffled to generate a random symbol table, specifically including:

[0018] Select a portion of the symbols from the original symbol table, shuffle their order, and leave the rest as is to generate a random symbol table;

[0019] Alternatively, the entire original symbol table can be shuffled to generate a random symbol table.

[0020] Furthermore, when the size of the selected new symbol table is larger than that of the original symbol table, a one-to-many mapping is formed. At this time, the frequency of the symbols in the corpus is counted, and the medium and high frequency symbols are preferentially mapped to multiple new symbols, so that the frequency of each symbol in the final mapping result is relatively average, and frequency analysis attacks cannot occur.

[0021] Furthermore, when the new symbol table is larger than the original symbol table, forming a one-to-many mapping, a random seed number s is added in each encryption process, so that the process of the one-to-many mapping is determined by the hash function h(s), thereby making the integrity of the encryption result verified by s.

[0022] Furthermore, when obtaining the original symbol table, if it is desired to apply the encrypted ciphertext m to a trained deep learning model, the vocabulary of the existing deep learning model is obtained, and the symbols automatically added by the word segmentation algorithm are removed to generate the original symbol table; if it is desired to apply the encrypted ciphertext m to an untrained deep learning model, the original symbol table is generated directly using the full vocabulary of the encrypted data.

[0023] Furthermore, the word segmentation algorithm is a character-based word segmentation algorithm.

[0024] Furthermore, the word segmentation algorithm used is a word-based segmentation algorithm.

[0025] An application of ciphertext generated by an encryption method in deep learning model training includes:

[0026] (1) Map each symbol in the symbol encoding table of the deep learning model into an encrypted symbol according to the mapping rules of key k;

[0027] (2) Use the ciphertext m to train or predict the deep learning model.

[0028] Furthermore, after mapping each symbol in the symbol encoding table of the deep learning model into an encrypted symbol according to the mapping rule of key k, the vector is perturbed in the symbol lookup table to improve the security of the ciphertext.

[0029] The beneficial effects of this invention are as follows:

[0030] (1) The encryption algorithm of the present invention does not require changes to the structure and computational logic of the deep learning model. Therefore, there is no additional computational overhead and no loss of model accuracy.

[0031] (2) The encryption algorithm of the present invention is concise and easy to implement, not limited by development language and development framework, and has strong applicability.

[0032] (3) The encryption algorithm of the present invention has high data encryption efficiency, reaching the level of 0.3 milliseconds, and is easier to deploy in online environments.

[0033] (4) When the mapping relationship is one-to-many, the present invention also provides the option to increase randomness and encryption verification, which can verify the integrity of encrypted text and deal with frequency analysis attacks. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0035] Figure 1 This is a flowchart illustrating the key generation process of the homomorphic encryption method for deep learning models in natural language processing according to the present invention.

[0036] Figure 2 This is a flowchart of the encryption process of the present invention.

[0037] Figure 3 This is a flowchart of the decryption process of the present invention.

[0038] Figure 4 This is a schematic diagram showing the original symbol frequency and the number of symbols mapped to a one-to-many relationship. Detailed Implementation

[0039] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0041] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0042] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0043] The homomorphic encryption method for deep learning models used in natural language processing according to the present invention includes two steps: key generation and data encryption.

[0044] Step 1: Key Generation

[0045] like Figure 1 As shown, an original symbol table is obtained based on the data to be encrypted or the deep learning model to be used; a random symbol table without duplicate symbols is generated based on the original symbol table, the size of the random symbol table is not smaller than the original symbol table, and the symbols in the original symbol table and the symbols in the random symbol table form a one-to-one or one-to-many mapping relationship; the random symbol table serves as the key K.

[0046] Generally, the symbol table used for deep learning may contain special symbols such as start, end, unknown, and padding. These symbols do not exist in the plaintext; they are automatically added to the text by the symbol decoding system. These special symbols are mapped as is, without needing to be randomly mapped to other symbols. That is, when obtaining the original symbol table, if it is desired to apply the encrypted ciphertext m to a trained deep learning model, the vocabulary of the existing deep learning model is obtained, and the symbols automatically added by the word segmentation algorithm are removed to generate the original symbol table. If it is desired to apply the encrypted ciphertext m to an untrained deep learning model, the original symbol table is generated directly using the full vocabulary of the encrypted data.

[0047] The key can be generated by completely shuffling the original symbol table to create a random symbol table; or by shuffling some symbols while keeping the rest intact. In this case, a one-to-one mapping relationship is formed between the symbols in the original symbol table and the symbols in the random symbol table. Alternatively, the key can be generated by selecting a completely new symbol table of a size no smaller than the original symbol table, randomly shuffling it, and creating a random symbol table. However, it is essential to ensure that no symbols are repeated, thus guaranteeing a one-to-one or one-to-many mapping and preventing many-to-one relationships. Such a mapping as a whole constitutes a symmetric key, which can be used for decryption and encryption.

[0048] When the size of the new symbol table is larger than that of the original symbol table, a one-to-many mapping is formed. At this time, the frequency of the symbols in the corpus is counted, and the medium and high frequency symbols are preferentially mapped to multiple new symbols, so that the frequency of each symbol in the final mapping result is relatively average, and frequency analysis attacks cannot occur.

[0049] When the new symbol table is larger than the original symbol table, forming a one-to-many mapping, a random seed number s is added during each encryption process. This ensures that the one-to-many mapping process is determined by the hash function h(s), thus allowing the integrity of the encryption result to be verified by s. Finally, the new symbols are concatenated, and s is stored as a check bit of the ciphertext. In this way, the ciphertext also achieves the function of verifying integrity. That is, an attacker cannot piece together two different ciphertext fragments to disguise them as new ciphertext, because the random seed numbers of different ciphertexts are likely to be different, such as s1, s2, etc. Therefore, h(s1, v) and h(s2, v) will obviously output different new symbols, and it is easy to find in the verification environment that these two do not come from the same message.

[0050] It is recommended to use a one-to-many mapping method to avoid attacks such as frequency analysis that can be achieved with one-to-one mapping. The specific mapping method is as follows:

[0051] When generating a one-to-many mapping, let the original symbol table be V and the new symbol table be W. |W| represents the size of W and satisfies |W|>|V|.

[0052] Establish a massive text database and count the number of times the symbol V appears in the text database, denoted as (f1, f2, ..., f...). n Then calculate the estimated number of mappings in the corresponding new symbol table:

[0053]

[0054] This value represents the estimated number of symbols mapped. However, the actual number of symbols in the new symbol table corresponding to this symbol should be an integer, not a decimal. Therefore, the rounding method is as follows:

[0055] make

[0056]

[0057]

[0058] Where w i If it is a pure decimal, then z i That is, 1, to ensure that symbol i can be mapped to at least one new symbol. Otherwise, if w i If it is greater than or equal to 1, then take z. i For w i The integer part, and the remaining decimal part is x. i .

[0059] Define r = |W| - ∑z i x i Arranged in descending order, the r-th element is called... but

[0060]

[0061] It refers to the number of symbols i that are mapped to the new symbol table.

[0062] It can be verified. In other words, this is a full mapping.

[0063] On the other hand, if the input is a string of text whose symbol frequency distribution is roughly the same as that in the original text library, then after such a one-to-many mapping, the frequency proportion of the most frequent symbol will be... Compared to the frequency of the average symbol, the difference in frequency is no more than twice. At this point, performing frequency analysis attacks on the ciphertext becomes very difficult.

[0064] Step 2: Data encryption.

[0065] Step (2.1): Divide the plaintext data into symbol strings using a word segmentation algorithm; the word segmentation algorithm can be either a word-based algorithm or a character-based algorithm.

[0066] Step (2.2): Map the symbols in the symbol string from the original symbols to new symbols according to the key K; wherein, if the mapping relationship is a one-to-one mapping, it is directly mapped to the new symbol; if the mapping relationship is a one-to-many mapping, a symbol is randomly selected from the multiple symbols mapped from the original symbol as the new symbol.

[0067] Step (2.3): Concatenate the new symbols to generate ciphertext m encrypted using key K. For example... Figure 2 As shown.

[0068] The text encrypted using the method of this invention can achieve homomorphic computation in deep learning models for natural language processing. In other words, training a model using encrypted text can achieve the same effect and computational efficiency as training a model using the original text.

[0069] The security analysis of the one-to-many mapping strategy is as follows:

[0070] For typical substitution ciphers, one attack method involves comparing the frequency of characters in the original text with the frequency of each new symbol in the ciphertext. This method employs a one-to-many symbol mapping, randomly mapping high-frequency symbols to one of multiple symbols, thus diluting the frequency of new symbols. The relationship between the number of new symbols and security is analyzed as follows:

[0071] In massive amounts of text, the statistical symbol w i Frequency of occurrence f i . w i Arranged from high frequency to low frequency, generally the word "w" i The ordinal number of a number is inversely proportional to its frequency, such as... Figure 4 As shown, that is to say:

[0072]

[0073] Without loss of generality, we can approximate |W| with 2n-1. Then, for the i-th symbol, we map it to... On a symbol, such as Figure 2 The bars in the diagram are shown. Therefore, the final total required is...

[0074]

[0075] A new symbol table. In other words, the size of the new symbol table is n times that of the original symbol table, which is log2|W| times the size. At this point, there exists a constant c such that the frequency of any symbol in the new symbol table is:

[0076]

[0077] because Therefore there is Therefore, it can be concluded that That is, the fluctuation range of any symbol frequency will not exceed 2 times.

[0078] like Figure 3 As shown, the integrity verification using a one-to-many mapping strategy is as follows:

[0079] When a one-to-many mapping exists, suppose it needs to be mapped to {w1, w2, ..., w...} nFor a set of size n, we can design a hash function whose domain is integers or subsets thereof, and whose range is {1, 2, ..., n}. For example, we can choose a prime number q and a large integer N such that N >> q >> n. Then, for a randomly sampled s from {1, 2, ..., N}, the function h(s) = (s mod q) mod n is such a function. Generally, given only h(s) and q unknown, if the probability of constructing another s' such that h(s') = h(s) does not exceed 1 / n, then this hash function is collision-resistant. The above is an example of a hash function; other construction methods are numerous.

[0080] When encrypting a message, a sample 's' is selected beforehand. All one-to-many mappings during the encryption process use this 's' to calculate the hash function, and the new symbol to be output is selected based on the hash function's value. The final ciphertext outputs 's' as a checksum, similar to a modern SHA or MD5 value. This way, when decrypting the message, the decryptor can compare each symbol in the decryption result to see if it can be encrypted using 's' to obtain the ciphertext, thus determining whether the message was encrypted from the same source, rather than being pieced together from multiple ciphertexts.

[0081] Generally, deep learning models process text in two ways: encoding symbols into IDs and converting IDs into embedding vectors. Therefore, the transfer learning process first requires mapping each symbol in the symbol encoding table to an encrypted symbol according to the mapping rules of the key mapping table.

[0082] Specifically, it can be divided into two situations:

[0083] The first scenario involves training a randomly initialized model directly from a massive amount of text. In this case, all the text needs to be encrypted before training, and the training method is the same as training with ordinary, unencrypted text.

[0084] The second scenario involves a pre-trained model trained on a large corpus that needs to be transferred to a model capable of reading ciphertext. In this case, only the symbol encoding algorithm and the symbol embedding layer logic of the model's input layer need to be transferred to the encryption model. The symbols in the symbol lookup table are modified to map them to their corresponding ciphertext. If it's a one-to-one mapping, it can be done directly. If it's a one-to-many mapping, such as mapping one symbol to N symbols, the data in the original table needs to be repeated N times to ensure that each newly mapped symbol corresponds to a vector of the original symbol in the lookup table.

[0085] After mapping is complete, the encrypted model can continue to be fine-tuned using encrypted data, or it can directly perform prediction tasks.

[0086] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A homomorphic encryption method for a deep learning model for natural language processing, characterized by, Comprising the following steps: Step one: key generation; According to the data to be encrypted or the deep learning model to be used, an original symbol table is obtained; a random symbol table without repeated symbols is generated according to the original symbol table, the size of the random symbol table is not less than that of the original symbol table, and the symbols in the original symbol table and the symbols in the random symbol table form a one-to-one or one-to-many mapping relationship; the random symbol table is used as a key K; Step two: data encryption; Step (2.1): a piece of plaintext data is divided into symbol strings through a word-based segmentation algorithm; Step (2.2): the symbols in the symbol string are mapped from the original symbols to new symbols according to the key K; if the mapping relationship is one-to-one mapping, it is directly mapped to the new symbol; if the mapping relationship is one-to-many, a symbol is randomly selected from the multiple symbols mapped from the original symbol as the new symbol; Step (2.3): the new symbol string is concatenated to generate the ciphertext m encrypted by the key K; When obtaining the original symbol table, if it is desired to apply the encrypted ciphertext m to the trained deep learning model, the vocabulary table of the existing deep learning model is obtained, and after removing the symbols automatically added by the segmentation algorithm, the original symbol table is generated; if it is desired to apply the encrypted ciphertext m to the untrained deep learning model, the original symbol table is directly generated by using the full amount of vocabulary table of the encrypted data. 2.The homomorphic encryption method for a deep learning model for natural language processing according to claim 1, wherein, According to the original symbol table, a random symbol table without repeated symbols is generated, which specifically includes: The order of the original symbol table is randomly disturbed to generate a random symbol table; at this time, the symbols in the original symbol table and the symbols in the random symbol table form a one-to-one mapping relationship; Or, a brand new symbol table with a size not less than the original symbol table is selected, and the order is randomly disturbed to generate a random symbol table. 3.The homomorphic encryption method for deep learning model of natural language processing according to claim 2, characterized in that, The order of the original symbol table is randomly disturbed to generate a random symbol table, which specifically includes: Part of the symbols in the original symbol table are selected to disturb the order, and the remaining symbols are kept original to generate a random symbol table; Or, the entire original symbol table is disturbed in order to generate a random symbol table. 4.The homomorphic encryption method for deep learning model of natural language processing according to claim 2, characterized in that, When the size of the selected new symbol table is greater than that of the original symbol table, a one-to-many mapping is formed; at this time, the frequency of the symbols in the corpus is counted, and the high-frequency symbols are preferentially mapped into multiple new symbols, so that the frequency of each symbol in the final mapping result is relatively average, and frequency analysis attack cannot occur. 5.The homomorphic encryption method for deep learning model of natural language processing according to claim 2, characterized in that, When the new symbol table is greater than the original symbol table, a one-to-many mapping is formed, and a random seed number s is added in each encryption process, so that the one-to-many mapping process is determined by the hash function h(s), and the integrity of the encryption result is verified by s.

6. An application method of ciphertext generated by the encryption method of claim 1 in deep learning model training, characterized in that, It includes: (1) Each symbol in the symbol encoding table of the deep learning model is mapped into an encrypted symbol according to the mapping rule of the key k; (2) The ciphertext m is used to train or predict the deep learning model.

7. The method of use of claim 6, wherein, After each symbol in the symbol encoding table of the deep learning model is mapped into an encrypted symbol according to the mapping rule of the key k, the vector is disturbed in the symbol lookup table to improve the security of the ciphertext.

Images