Speech feature random convolution speech information transmission method based on homomorphic encryption

Abstract

The present invention discloses a method for transmitting voice information based on homomorphic encryption of voice features in disordered convolution, which relates to the field of artificial intelligence and computer security technology. The method comprises: step 1, obtaining a voice feature vector; step 2, scrambling the extracted voice feature vector; step 3, generating a matrix with sequence information of the scrambled feature vector; step 4, the user converts the above-obtained disordered feature vector X shuffled The server provides the matrix C to the service provider, and instructs the service provider to complete the convolution in an encrypted state; Step 5: The service provider returns the calculated convolution result to the user in the form of ciphertext, and the user receives the result and decrypts it with a private key to obtain the final result. The present invention realizes the convolution operation without leaking the sequence information, ensures the security and reliability of the data transmission and processing process, effectively avoids the risk of sensitive data leakage, and improves the flexibility of speech features in different application scenarios.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and computer security technology, and in particular to a method for transmitting speech information based on homomorphic encryption and random convolution of speech features. Background Art

[0002] With the advent of the 5G era, the rapid development of the communications industry, and the high popularity of digitalization, speech processing technology has been widely used in various fields, such as speech recognition, speech synthesis, and sound processing. Speech features are mathematical representations of speech signals. In the processing of related tasks, convolutional neural networks (CNNs) are often used to operate them. However, existing speech feature processing methods rely on specific sequential information, which means that once the corresponding sequential relationship is destroyed or changed, the convolution result may be distorted, which may affect the accuracy and robustness of the entire speech processing system. Therefore, this dependence greatly limits the flexible application of speech features in different scenarios.

[0003] At the same time, the privacy protection of voice data is also a highly concerned issue: a large amount of private data is involved in the process of voice recognition, synthesis and processing. Malicious cloud service providers may obtain sensitive information such as the target speaker's voiceprint and even the content of the speech from the corpus uploaded by the user, which may cause multiple negative effects such as personal information data leakage, the production of bad information audio for broadcasting and illegal activities such as fraud, and the creation of bad public opinion to disrupt social order. In order to avoid the above risks, appropriate encryption methods are necessary. Homomorphic encryption technology, based on its advantage of being able to achieve "calculated invisible" data, has become one of the research hotspots in related fields in recent years. It allows data to be calculated in an encrypted state, and the results obtained are consistent with the results of the same calculation in the plaintext state. This feature also makes it possible to convolve voice features without decrypting the data, so it can effectively avoid data leakage.

[0004] Therefore, there are two main problems with existing speech feature processing methods: first, the reliance on sequential information leads to limited application flexibility, and second, insufficient privacy protection of speech data. In view of the above problems, when processing speech data, it is particularly important to reasonably utilize existing technologies to improve the flexibility and breadth of their application while ensuring their security.

[0005] Therefore, it is an urgent problem for technical personnel in this field to propose a method for transmitting speech information by random convolution of speech features based on homomorphic encryption to solve the difficulties existing in the prior art. Summary of the invention

[0006] In view of this, the present invention provides a method for transmitting speech information by random convolution of speech features based on homomorphic encryption, which can achieve the goal of correctly completing speech feature convolution without leaking sequence information through homomorphic encryption technology, speech processing, algorithms and data processing.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A method for transmitting speech information based on homomorphic encryption and random convolution of speech features comprises the following steps:

[0009] S1, feature vector acquisition step: extracting speech feature vector X from speech information;

[0010] S2, random feature vector generation step: reorder the extracted speech feature vector X to obtain a random feature vector X shuffled ;

[0011] S3. Matrix generation step: Generate the matrix C = A·M required for convolution T , where A is the parameter matrix and M is the random eigenvector X shuffled Matrix of sequential information;

[0012] S4, convolution step: The user converts the disordered feature vector X in S2 shuffled The matrix C required for the convolution in S3 is sent to the service provider, which is asked to complete the convolution in an encrypted state and obtain the convolution result in an encrypted state;

[0013] S5, return step: The service party returns the calculated convolution result to the user party in the form of ciphertext. After the user party accepts it, it decrypts it with the private key to obtain the final result.

[0014] In the above method, optionally, the specific steps in S2 include:

[0015] S201, determine the original speech feature vector: the original speech feature vector is represented as X = [x1, x2, x3, ... x i …, x n ] T , where xi is the feature value of the vector in the i-th frame on the time axis, and n is the total number of frames;

[0016] S202, generating a randomly arranged permutation sequence P, where P[i] represents the i-th element in the permutation sequence, and also represents the sequential position of the feature quantities in the i-th frame after being disrupted;

[0017] S203, updating the order information of the original speech feature vector X according to the created permutation sequence P, that is, rearranging the feature quantities in each time frame of the original speech feature vector X according to the order of the integers in the permutation sequence P, and obtaining a shuffled sequence X shuffled =[x P[1] ,,x p[2] , x p[3] , …x p[i] …, x p[n] ] T .

[0018] In the above method, optionally, the specific steps in S3 include:

[0019] S301. Assume that the convolution kernel of 1×(2m-1) is [a1, a2, ···, a 2m-1 ] (1≤m≤n), where a1, a2, …, a 2m-1 The specific value of should be obtained by the service provider through training, and the parameter matrix A is generated to obtain the weighted average of the eigenvalues ​​of the speech feature vector on consecutive (2m-1) frames;

[0020] S302, generate a random feature vector X shuffled The matrix M of the order information;

[0021] S303, generate the matrix C required for convolution, matrix C and X shuffled The convolution result is equal to the result obtained by convolving the parameter matrix A with the original speech feature vector X:

[0022] X shuffled =M·X

[0023] M T M=E

[0024] A·X=A·(M T ·M)·X=A·M T ·(M·X)=A·M T ·X shuffled .

[0025] The above method, optionally, S301 specifically includes:

[0026] (1) Generate 2m-1 special matrices of size n×n whose elements contain only 0 and 1, denoted as A1, A2, …, A 2m-1 ;

[0027] (2) Let a1, a2, …, a 2m-1 Respectively with A1, A2, ..., A 2m-1 Multiply and add the results to get the required matrix A.

[0028] The above method, optionally, S302 specifically includes:

[0029] (1) Generate an n×n identity matrix M units ;

[0030] (2) Based on (1), the matrix M is transformed into units The row vectors of are exchanged accordingly to obtain the required matrix M corresponding to the disordered vector, that is, M units Replace the i-th row of with the P[i]th row:

[0031] M[i,:]=M units [P[i],:]

[0032] Get the eigenvector X with random order shuffled The matrix M of the order information.

[0033] In the above method, optionally, the specific steps in S4 include:

[0034] S401, encryption part: the user generates the key Key required for encrypting data by running the key generation function KeyGen, and uses homomorphic encryption technology to encrypt the matrix C required for convolution, and obtains C ciphertext ;

[0035] S402, transmission part: the user sends the generated disordered feature vector X shuffled And the processed ciphertext C ciphertext Send to the service provider;

[0036] S403, convolution calculation: the service provider completes the convolution operation of the matrix in the ciphertext state and obtains the convolution result in the encrypted state.

[0037] In the above method, optionally, the homomorphic encryption technology in S401 includes but is not limited to:

[0038] The CKKS scheme, GSW scheme, BV scheme in fully homomorphic encryption and the Paillier algorithm and ElGamal algorithm in partially homomorphic encryption.

[0039] It can be seen from the above technical solution that, compared with the prior art, the present invention provides a speech feature random convolution speech information transmission method based on homomorphic encryption, which has the following beneficial effects:

[0040] 1. Independent of sequence information: The present invention can realize correct convolution operation in the case of uncertain sequence or time sequence by encrypting the disordered speech feature vector and the matrix with its sequence information, and on this basis, the flexibility and applicability of speech feature processing are guaranteed;

[0041] 2. Protecting data privacy: The present invention introduces homomorphic encryption technology to encrypt voice features, so that the data is encrypted during transmission and processing. This effectively protects the privacy of voice data and reduces the risk of data leakage, which is especially important in scenarios involving personal privacy such as voice recognition and dialogue systems.

[0042] 3. Broaden the application areas: Traditional speech feature processing methods are limited by their reliance on sequential information and may not be able to adapt to complex and changeable application scenarios; the method proposed in the present invention can be used in a wider range of application areas such as speech synthesis (TTS, Text-to-Speech), emotion recognition, speaker recognition, etc. Take TTS as an example for specific explanation. The TTS system aims to convert text into natural and fluent speech output, and is the basis for applications such as human-computer interaction and intelligent assistants. The method proposed in the present invention can make the system more flexible in processing texts of different lengths and expressions, and can provide a solid foundation for further optimization of steps such as model reasoning and target synthesis, which plays an extremely important role in improving the accuracy and fluency of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0044] Figure 1 This is a flow chart of the method for transmitting speech information using random convolution of speech features based on homomorphic encryption disclosed in the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0047] Reference Figure 1 As shown, the present invention discloses a method for transmitting speech information based on homomorphic encryption and random convolution of speech features, comprising the following steps:

[0048] S1, feature vector acquisition step: extracting speech feature vector X from speech information;

[0049] S2, random feature vector generation step: reorder the extracted speech feature vector X to obtain a random feature vector X shuffled ;

[0050] S3. Matrix generation step: Generate the matrix C = A·M required for convolution T , where A is the parameter matrix and M is the random eigenvector X shuffled Matrix of sequential information;

[0051] S4, convolution step: The user converts the disordered feature vector X in S2 shuffled The matrix C required for the convolution in S3 is sent to the service provider, which is asked to complete the convolution in an encrypted state and obtain the convolution result in an encrypted state;

[0052] S5, return step: The service party returns the calculated convolution result to the user party in the form of ciphertext. After the user party accepts it, it decrypts it with the private key to obtain the final result.

[0053] Furthermore, the specific steps in S2 include:

[0054] S201, determine the original speech feature vector: the original speech feature vector is represented as X = [x1, x2, x3, ... x i …,x n ] T , where x i is the feature value of the vector in the i-th frame on the time axis, and n is the total number of frames;

[0055] S202, generating a randomly arranged permutation sequence P, where P[i] represents the i-th element in the permutation sequence, and also represents the sequential position of the feature quantities in the i-th frame after being disrupted;

[0056] S203, updating the order information of the original speech feature vector X according to the created permutation sequence P, that is, rearranging the feature quantities in each time frame of the original vector according to the order of the integers in the permutation sequence P, to obtain a shuffled sequence X shuffled =[x P[1] ,x p[2] ,x p[3] ,…x p[i] …,x p[n] ] T .

[0057] Specifically, for the convenience of examples and calculations, let the size of X be 2×8, assuming

[0058]

[0059] Let the permutation sequence P = [6, 3, 1, 4, 8, 5, 2, 7], and rearrange the original speech feature vector X according to the order in the sequence P to obtain the shuffled feature vector:

[0060]

[0061] Furthermore, the specific steps in S3 include:

[0062] S301, assuming that the convolution kernel of 1×(2m-1) is [a1,a2,…,a 2m-1 ] (1≤m≤n), where a1, a2, …, a 2m-1 The specific value of should be obtained by the service provider through training, and the parameter matrix A is generated to obtain the weighted average of the eigenvalues ​​of the speech feature vector on consecutive (2m-1) frames;

[0063] S302, generate a random feature vector X shuffled The matrix M of the order information;

[0064] S303, generate the matrix C required for convolution, matrix C and X shuffled The convolution result is equal to the result obtained by convolving the parameter matrix A with the original speech feature vector X:

[0065] X shuffled =M·X

[0066] M T M=E

[0067] A·X=A·(M T ·M)·X=A·M T·(M·X)=A·M T ·X shuffled .

[0068] Furthermore, S301 specifically includes:

[0069] (1) Generate 2m-1 special matrices of size n×n whose elements contain only 0 and 1, denoted as A1, A2, …, A 2m-1 ;

[0070] Specifically,

[0071]

[0072] Among them, A m It is a diagonal matrix, that is, all elements on the diagonal are 1, and the rest are 0;

[0073] A m-1 : The elements on the first diagonal line below the left diagonal are all 1, and the rest are 0;

[0074] A m+1 : The elements on the first diagonal line to the upper right of the diagonal are all 1, and the rest are 0;

[0075] And so on A m-a This means that the elements on the ath diagonal line to the lower left of the diagonal are all 1, and the rest are 0; A m+a This means that all elements on the ath diagonal line to the upper right of the diagonal are 1, and the rest are 0.

[0076] (2) Let a1, a2, …, a 2m-1 Respectively with A1, A2, ..., A 2m-1 Multiply and add the results to get the required matrix A.

[0077] Specifically, A=a1A1+…+a m A m +…+a 2m-1 A 2m-1

[0078]

[0079]

[0080] Here, it is assumed that the convolution kernel size is 1×3, and the specific values ​​are [a1, a2, a3] = [0.5, 0.3, 0.2]. After substituting into the calculation steps in S301, we can get:

[0081]

[0082] Furthermore, S302 specifically includes:

[0083] (1) Generate an n×n identity matrix M units ;

[0084] (2) Based on (1), the matrix M is transformed into units The row vectors of are exchanged accordingly to obtain the required matrix M corresponding to the disordered vector, that is, M units Replace the i-th row of with the P[i]th row:

[0085] M[i,:]=M units [P[i],:]

[0086] Get the eigenvector X with random order shuffled The matrix M of the order information.

[0087] Specifically, if the jth column of the i-th row vector of the shuffled matrix after the row transformation is 1, it means that the i-th frame of the shuffled feature vector was originally the j-th frame.

[0088] According to the permutation sequence P = [6, 3, 1, 4, 8, 5, 2, 7], M units The i-th row of M is replaced by the P[i]th row to obtain M shuffled :

[0089]

[0090] S303, generate the matrix C required for convolution, which is shuffled The convolution of is the same as the convolution of A directly with X:

[0091] X shuffled =M·X

[0092] M T M=E

[0093] A·X=A·(M T ·M)·X=A·M T ·(M·X)=A·M T ·X shuffled .

[0094] Specifically, the following verification is performed:

[0095]

[0096] A and X, C and X respectively shuffled Multiplying is equivalent to adding X or X shuffled Convolve with the convolution kernel of [0.5, 0.3, 0.2] and determine whether the results are equal:

[0097]

[0098]

[0099] The calculation results of the two are equal, that is, the ideal effect can be achieved. The convolution value calculated after scrambling is exactly the same as the value without scrambling.

[0100] Furthermore, the specific steps in S4 include:

[0101] S401, encryption part: the user generates the key Key required for encrypting data by running the key generation function KeyGen, and uses homomorphic encryption technology to encrypt the matrix C required for convolution, and obtains C ciphertext ;

[0102] S402, transmission part: the user sends the generated disordered feature vector X shuffled And the processed ciphertext C ciphertext Send to the service provider;

[0103] S403, convolution calculation: the service provider completes the convolution operation of the matrix in the ciphertext state and obtains the convolution result in the encrypted state.

[0104] Furthermore, the homomorphic encryption technology in S401 should be able to ensure convolution homomorphism, including but not limited to:

[0105] The CKKS scheme, GSW scheme, BV scheme in fully homomorphic encryption and the Paillier algorithm and ElGamal algorithm in partially homomorphic encryption.

[0106] Specifically, the CKKS (Cheon-Kim-Kim-Song) scheme, GSW (Gentry-Sahai-Waters) scheme, BV (Brakerski-Vaikuntanathan) scheme in fully homomorphic encryption (FHE), and the Paillier algorithm and ElGamal algorithm in partially homomorphic encryption (PHE).

[0107] The current existing speech feature processing methods rely on the specific order information of the speech feature vector, which means that once the order relationship is destroyed or changed, it may lead to the inability to obtain the correct convolution result, thus affecting the accuracy and robustness of the entire speech processing system, which also greatly limits the flexible application of speech features in different scenarios. The present invention can realize the convolution operation without leaking the order information, ensure the security and reliability of the data transmission and processing process, effectively avoid the risk of sensitive data leakage, eliminate the dependence of speech processing on order information, and provide a basis for further optimization of model reasoning, target synthesis and other steps. It plays an extremely important role in improving the accuracy and fluency of speech synthesis, and can be widely used in speech synthesis (TTS, Text-to-Speech), emotion recognition, speaker recognition and other fields.

[0108] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0109] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for transmitting speech information based on homomorphic encryption and random convolution of speech features, characterized in that: The following steps are involved: S1, feature vector acquisition step: extracting speech feature vector X from speech information; S2, random feature vector generation step: reorder the extracted speech feature vector X to obtain a random feature vector X shuffled ; S3. Matrix generation step: Generate the matrix C = A·M required for convolution T , where A is the parameter matrix and M is the random eigenvector X shuffled Matrix of sequential information; S4, convolution step: The user converts the disordered feature vector X in S2 shuffled The matrix C required for the convolution in S3 is sent to the service provider, which is asked to complete the convolution in an encrypted state and obtain the convolution result in an encrypted state; S5, return step: the service party returns the calculated convolution result to the user party in the form of ciphertext. After the user party receives it, it decrypts it with the private key to obtain the final result; The specific steps in S2 include: S201, determine the original speech feature vector: the original speech feature vector is represented as X = [x1, x2, x3, ... x i …,x n ] T , where x i is the feature value of the vector in the i-th frame on the time axis, and n is the total number of frames; S202, generating a randomly arranged permutation sequence P, where P[i] represents the i-th element in the permutation sequence, and also represents the sequential position of the feature quantities in the i-th frame after being disrupted; S203, updating the order information of the original speech feature vector X according to the created permutation sequence P, that is, rearranging the feature quantities in each time frame of the original vector according to the order of the integers in the permutation sequence P, to obtain a shuffled sequence X shuffled =[x P[1], x p[2] ,x p[3] ,…,x p[i] …,x p[n] ] T .

2. According to the method for transmitting speech information based on homomorphic encryption of speech features in random order convolution, the method is characterized in that: The specific steps in S3 include: S301, assuming that the convolution kernel of 1×(2m-1) is [a1,a2,…,a 2m-1 ](1≤m≤n), where a1, a2, …, a 2m-1 The specific value of should be obtained by the service provider through training, and the parameter matrix A is generated to obtain the weighted average of the eigenvalues ​​of the speech feature vector on consecutive (2m-1) frames; S302, generate a random feature vector X shuffled The matrix M of the order information; S303, generate the matrix C required for convolution, matrix C and X shuffled The convolution result is equal to the result obtained by convolving the parameter matrix A with the original speech feature vector X: X shuffled =M·X M T ·M=E A·X=A·(M T ·M)·X=A·M T ·(M·X)=A·M T ·X shuffled 。 3. According to claim 2, a method for transmitting voice information based on homomorphic encryption of voice features in random order convolution is characterized in that: S301 specifically includes: (1) Generate 2m-1 special matrices of size n×n whose elements contain only 0 and 1, denoted as A1, A2, …, A 2m-1 ; (2) Let a1, a2, …, a 2m-1 Respectively with A1, A2, ..., A 2m-1 Multiply and add the results to get the required matrix A.

4. According to a method for transmitting speech information based on homomorphic encryption of speech features in random order convolution, the method is characterized in that: S302 specifically includes: (1) Generate an n×n identity matrix M units ; (2) Based on (1), the matrix M is transformed into units The row vectors of are exchanged accordingly to obtain the required matrix M corresponding to the disordered vector, that is, M units Replace the i-th row of with the P[i]th row: M[i,:]=M units [P[i],:] Get the eigenvector X with random order shuffled The matrix M of the order information.

5. According to a method for transmitting speech information based on homomorphic encryption of speech features in random order convolution, the method is characterized in that: The specific steps in S4 include: S401, encryption part: the user generates the key Key required for encrypting data by running the key generation function KeyGen, and uses homomorphic encryption technology to encrypt the matrix C required for convolution, and obtains C ciphertext ; S402, transmission part: the user sends the generated disordered feature vector X shuffled And the processed ciphertext C ciphertext Send to the service provider; S403, convolution calculation: the service provider completes the convolution operation of the matrix in the ciphertext state and obtains the convolution result in the encrypted state.

6. According to claim 5, a method for transmitting voice information based on homomorphic encryption of voice features in random order convolution is characterized in that: Homomorphic encryption technologies in S401 include but are not limited to: The CKKS scheme, GSW scheme, BV scheme in fully homomorphic encryption and the Paillier algorithm and ElGamal algorithm in partially homomorphic encryption.

Images