A method and system for encrypted domain speech recognition based on complex neural network

Abstract

The present invention proposes an encrypted domain speech recognition method and system based on a complex neural network, which relates to the technical field of speech recognition. A speech data set is used to train a test model, and the trained model is encapsulated in a speech recognition execution server for standby; a privacy server generates a public key and a private key and sends them to a client, and sends the public key to the speech recognition execution server. The client encodes and encrypts the speech data and sends it to the speech recognition execution server. The speech recognition execution server recognizes the encrypted speech in the encrypted domain using the model. The adaptability and matching of the complex neural network model facilitates the speech recognition execution server to efficiently recognize the encrypted speech in the encrypted domain, and sends the recognition result to the client. The client decrypts and obtains the speech recognition result. While ensuring privacy and security, the voice owner of the client does not need to perform feature extraction, and can recognize multiple voices, thereby reducing the time overhead of unit encrypted speech recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of encrypted domain speech recognition, and more specifically, to an encrypted domain speech recognition method and system based on a complex neural network. Background Art

[0002] Speech recognition is one of the important topics in speech processing. However, speech recognition in cloud computing may cause privacy leakage. The existing encrypted signal processing technology (SPED) provides cloud servers with the ability to process large amounts of speech data while ensuring privacy security, so that cloud servers can perform tasks such as speech recognition, speech verification, and speaker recognition. From the perspective of data collection, users store private resources such as audio data required for speech recognition on the client. During speech recognition in the cloud environment, the client transmits private resources to the cloud server. In this process, since the data currently transmitted is encrypted data, the user's private data is often leaked by the cloud server, either actively or passively. The potential security risks brought about by this privacy data leakage are very serious and may be used for profit by others, or even captured by criminals and used for illegal activities.

[0003] At present, the CKKS cryptographic system is one of the most popular fully homomorphic encryption systems, and speech recognition technology based on homomorphic encryption is widely used. For example, in 2019, S. Zhang, Y. Gong, and DY u et al. published an article entitled "Encrypted speech recognition using deep polynomial networks" at the IEEE International Conference on Acoustics, Speech and Signal Processing 2019. The article pointed out that the voice owner encrypts the features extracted from the voice data through the public key, and then transmits it to the server. The server performs speech recognition in the ciphertext domain, and then recognizes the features extracted based on the encrypted domain neural network, and transmits it to the voice owner for decryption. Finally, the decrypted data is decoded to obtain the speech recognition result. Under this method, the server does not need to use a private key to perform encryption domain neural network operations, and the voice owner receives the private key decryption result through a secure channel, and the private key cannot be obtained by illegal adversaries, so that the privacy and security of the voice during the recognition process is guaranteed. However, on the one hand, the homomorphic encryption used in this article occupies a large amount of computing resources and the time overhead for unit encrypted voice data recognition is high; on the other hand, the voice owner needs to extract features from the local audio and then encrypt it, and decode the network output locally, which brings a certain computational burden to the voice owner. Summary of the invention

[0004] In order to solve the problem that the existing speech recognition method based on homomorphic encryption has high time overhead per unit of encrypted speech recognition and high computational burden on the voice owner during the speech recognition process, the present invention proposes an encrypted domain speech recognition method and system based on a complex neural network. While ensuring privacy and security, the voice owner only needs to perform simple encryption operations on the local audio without feature extraction, and can recognize multiple voice data at the same time, reducing the time overhead per unit of encrypted speech recognition.

[0005] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0006] A method for speech recognition in an encrypted domain based on a complex neural network, comprising:

[0007] S1. Collect a speech data set, divide the speech data set into a training set and a test set, and treat the speech data in the training set and the test set as plural;

[0008] S2. Construct a complex neural network model, train the complex neural network model using the training set, test the accuracy of the complex neural network model using the test set, obtain the final complex neural network model, and encapsulate it in the speech recognition execution server;

[0009] S3. Generate a public key pk and a private key sk using a privacy server, send both the public key pk and the private key sk to the client, and send the public key pk to the speech recognition execution server;

[0010] S4. The client uses the public key pk to encode and encrypt the voice to be recognized. Each voice to be recognized has several tags. Each voice data of the encrypted voice to be recognized is regarded as a plural number and sent to the voice recognition execution server;

[0011] S5. The plural neural network model in the speech recognition execution server performs speech recognition on the encrypted speech in the encryption domain, and obtains the encryption probability of several labels corresponding to the encrypted speech;

[0012] S6. The speech recognition execution server sends the encrypted probabilities of several tags to the client. After the client decrypts the tags using the private key sk, it takes the tag with the largest probability value as the result of speech recognition.

[0013] In this technical solution, a speech data set is first collected, and then a complex neural network model is constructed. The complex neural network model is trained and tested using the speech data set. The trained complex neural network model is encapsulated in a speech recognition execution server for standby use. The complex neural network model is adapted to match the encrypted speech form. In the speech encryption and decryption stage, a privacy server is used to generate a public key and a private key and send them to the client, and the public key is sent to the speech recognition execution server. Then the client encodes and encrypts the speech data and sends it to the speech recognition execution server. The speech recognition execution server uses the complex neural network model to recognize the encrypted speech in the encrypted domain. The adaptability and matching of the complex neural network model facilitates the speech recognition execution server to efficiently recognize the encrypted speech in the encrypted domain. Then the recognition result is sent to the client, and the client decrypts to obtain the speech recognition result. This solution allows the speech recognition execution server to recognize multiple voices at the same time, and the client does not need to extract features of the local audio. The data transmitted between the speech recognition server and the client are all ciphertext, and the encrypted data can only be decrypted by the voice owner, which meets the privacy requirements of both the speech recognition execution server and the client, ensuring that speech recognition is performed under privacy security.

[0014] Preferably, in step S1, each keyword in the voice data set is composed of a group of Z seconds of voice data from different voice sources, the voice data is provided with a label, and each voice data is labeled with one of several keywords.

[0015] Preferably, after collecting the speech data set, the speech data in the speech data set is sampled to make it speech data with a sampling frequency of Q and a length of Z seconds, matching the input dimension of the complex neural network model.

[0016] Preferably, the complex neural network model constructed in step S2 includes a one-dimensional convolutional network unit and a fully connected network unit, and the speech data is output as a one-dimensional feature vector through the one-dimensional convolutional network unit, and the one-dimensional feature vector is used as the input of the fully connected network unit; the one-dimensional convolutional network unit includes six convolution blocks, each of which is composed of a complex convolution layer, a complex activation layer and an average pooling layer, and the complex convolution layer splits the speech data input to the complex neural network model into a real part and an imaginary part, and a complex convolution kernel is designed in the complex convolution layer to simulate complex operations; assuming that the speech data input to the complex neural network model is M=X+iY, X and Y are the real part and imaginary part of the speech data M respectively, and the complex convolution kernel vector is K=A+iB, A and B are the real part and imaginary part of the complex convolution kernel, then the speech data M is operated through the complex convolution layer, and the output is:

[0017] M*K=(X*AY*B)+i(X*B+Y*A)

[0018] Converted into a form that conforms to the complex convolution kernel operation, expressed as;

[0019]

[0020] Among them, * represents the convolution operation, (M*K) real 、(M*K) imag Respectively represent the real and imaginary components of M*K;

[0021] The activation function of the complex activation layer is the PEReLU function or the PCReLU function. For the speech data C input into the complex neural network model, the expression of the PEReLU function is:

[0022] PEReLU(C)=EC 3 +FC 2 +GC+H

[0023] The expression of the PCReLU function is:

[0024]

[0025] Among them, C real , C imag Represent the real and imaginary parts of C respectively, E, F, G, H are the polynomial coefficients of the PEReLU function, E1, F1, G1, H1 are the polynomial coefficients of the real part of the PEReLU function, E2, F2, G2, H2 are the polynomial coefficients of the imaginary part of the PEReLU function;

[0026] The fully connected network unit includes a first complex fully connected layer, an activation function layer and a second complex fully connected layer. The activation function layer is located between the first complex fully connected layer and the second complex fully connected layer. The activation function of the activation function layer is a PEReLU function or a PCReLU function. The first complex fully connected layer and the second complex fully connected layer are both implemented by splitting complex vectors and simulating complex operations. The output of the second complex fully connected layer serves as the output layer of the fully connected network unit.

[0027] Here, the adaptive matching of the constructed complex neural network model and the encrypted speech form and the design of the activation function facilitate the subsequent speech recognition execution server to efficiently and accurately recognize the encrypted speech in the encrypted domain.

[0028] Preferably, when training the complex neural network model, the speech data of the training set is used as the input of the complex neural network model, and the labels corresponding to the speech data of the training set are used as the output labels of the complex neural network model. The loss function adopts cross entropy and is trained by gradient descent method.

[0029] Preferably, in step S4, the process of the client using the public key pk to perform data encoding and encryption on the speech to be recognized is:

[0030] Assume that there are n voice data in the speech to be recognized, represented by: 0 , v 1 , …, v n-1 , each voice data has m points: s=0,1,…,n-1,the formula is executed for all speech data:

[0031]

[0032] Wherein, j = 0, 1, ..., m-1;

[0033] Encode n voice data into m vectors V 0 , V 1 , …, V m-1 ,in:

[0034]

[0035] Use the public key pk to identify m vectors V 0 , V 1 , …, V m-1 Encrypt and get the encrypted data:

[0036] Preferably, in step S5, when the complex neural network model in the speech recognition execution server performs speech recognition on the encrypted speech in the encryption domain, the multiplication used is the homomorphic multiplication of the encryption domain.

[0037] Preferably, in step S5, when the complex neural network model in the speech recognition execution server performs speech recognition on the encrypted speech in the encryption domain, the addition used is the homomorphic addition of the encryption domain.

[0038] The present application also proposes an encrypted domain speech recognition system based on a complex neural network, comprising:

[0039] A data set collection unit, used for collecting a speech data set and dividing the speech data set into a training set and a test set, wherein the speech data in the training set and the test set are both regarded as plural;

[0040] A complex neural network construction training module is used to construct a complex neural network model, train the complex neural network model using a training set, test the accuracy of the complex neural network model using a test set, and obtain a final complex neural network model, which is located in the speech recognition execution server;

[0041] The privacy server is used to generate a public key pk and a private key sk, and send both the public key pk and the private key sk to the client, and send the public key pk to the speech recognition execution server;

[0042] The client uses the public key pk to encode and encrypt the speech to be recognized. Each speech to be recognized has several tags. Each voice data of the encrypted speech to be recognized is regarded as a plural number and sent to the speech recognition execution server.

[0043] The speech recognition execution server has a complex neural network construction training module, which performs speech recognition on the encrypted speech in the encrypted domain, obtains the encryption probabilities of several labels corresponding to the encrypted speech, and sends them to the client;

[0044] The client also uses the private key sk to decrypt the encrypted probabilities of several labels, and takes the label with the largest probability value as the result of speech recognition.

[0045] Preferably, the system also includes: a sampling processing module, after the data set acquisition unit acquires the speech data set, the sampling processing module samples the speech data in the speech data set to make it speech data with a sampling frequency of Q and a length of Z seconds, matching the input dimension of the complex neural network model.

[0046] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0047] The present invention proposes an encrypted domain speech recognition method and system based on a complex neural network. First, a speech data set is collected, and then a complex neural network model is constructed. The complex neural network model is trained and tested using the speech data set, and the trained complex neural network model is encapsulated in a speech recognition execution server for standby. In the speech encryption and decryption stage, a public key and a private key are generated by a privacy server and sent to a client, and the public key is sent to the speech recognition execution server. Then, the client encodes and encrypts the speech data and sends it to the speech recognition execution server. The speech recognition execution server recognizes the encrypted speech in the encrypted domain using the complex neural network model. The adaptability and matching of the complex neural network model facilitates the speech recognition execution server to efficiently recognize the encrypted speech in the encrypted domain. Then, the recognition result is sent to the client, and the client decrypts to obtain the speech recognition result. While ensuring privacy security, the voice owner of the client only needs to perform a simple encryption operation on the local audio without feature extraction. The speech recognition execution server can recognize multiple voices at the same time, thereby reducing the time overhead of unit encrypted speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram showing a flow chart of the encrypted domain speech recognition method based on a complex neural network proposed in Embodiment 1 of the present invention;

[0049] Figure 2 A structural diagram showing an encrypted domain speech recognition system based on a complex neural network proposed in Embodiment 3 of the present invention;

[0050] Figure 3 A structural diagram showing the second encrypted domain speech recognition system based on complex neural network proposed in Example 3 of the present invention. DETAILED DESCRIPTION

[0051] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0052] In order to better illustrate the present embodiment, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the actual size;

[0053] It is understandable to those skilled in the art that descriptions of certain well-known contents in the drawings may be omitted.

[0054] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0055] The positional relationships described in the drawings are only for illustrative purposes and should not be construed as limiting the present patent;

[0056] Example 1

[0057] In this embodiment, if Figure 1 As shown, a method for encrypted domain speech recognition based on a complex neural network is proposed, the method comprising:

[0058] S1. Collect a speech data set, divide the speech data set into a training set and a test set, and treat the speech data in the training set and the test set as plural;

[0059] S2. Construct a complex neural network model, train the complex neural network model using the training set, test the accuracy of the complex neural network model using the test set, obtain the final complex neural network model, and encapsulate it in the speech recognition execution server;

[0060] S3. Generate a public key pk and a private key sk using a privacy server, send both the public key pk and the private key sk to the client, and send the public key pk to the speech recognition execution server;

[0061] S4. The client uses the public key pk to encode and encrypt the voice to be recognized. Each voice to be recognized has several tags. Each voice data of the encrypted voice to be recognized is regarded as a plural number and sent to the voice recognition execution server;

[0062] S5. The plural neural network model in the speech recognition execution server performs speech recognition on the encrypted speech in the encryption domain, and obtains the encryption probability of several labels corresponding to the encrypted speech;

[0063] S6. The speech recognition execution server sends the encrypted probabilities of several tags to the client. After the client decrypts the tags using the private key sk, it takes the tag with the largest probability value as the result of speech recognition.

[0064] In general, the method proposed in this embodiment is actually divided into an offline stage and an online stage during the implementation process. The offline stage is a stage of constructing and training a plural neural network model for backup, corresponding to steps S1 to S2, and the online stage is a stage of performing online speech recognition using a plural neural network model trained offline, corresponding to steps S3 to S6. The scheme consisting of the offline stage and the online stage allows the speech recognition execution server to recognize multiple voices at the same time, and the client does not need to extract features of local audio. The data transmitted between the speech recognition server and the client are all ciphertext, and the encrypted data can only be decrypted by the voice owner, which meets the privacy requirements of both the speech recognition execution server and the client, ensuring that speech recognition is performed under privacy security.

[0065] In step S1, each keyword in the speech data set consists of a set of Z seconds of speech data from different speech sources, and the speech data is provided with labels, and each piece of speech data is labeled with one of several keywords. In this embodiment, the collected speech data set uses the SpeechCommands data set open sourced by the Google Brain team. Each keyword in the speech data set consists of a set of Z seconds of speech data from different speakers. A total of 105,829 pieces of speech data are used for training and prediction, and each piece of speech data is labeled with one of 35 keywords.

[0066] After collecting the speech data set, the speech data in the speech data set is sampled to become speech data with a sampling frequency of Q and a length of Z seconds, matching the input dimension of the complex neural network model. In this embodiment, the speech data is sampled to a length of 8000 Hz, and in this embodiment, Z is 1 second.

[0067] like Figure 2As shown, the constructed complex neural network model includes a one-dimensional convolutional network unit and a fully connected network unit. The speech data is output as a one-dimensional feature vector through the one-dimensional convolutional network unit, and the one-dimensional feature vector is used as the input of the fully connected network unit; the one-dimensional convolutional network unit includes six convolution blocks, each of which is composed of a complex convolution layer, a complex activation layer and an average pooling layer. The complex convolution layer splits the speech data input to the complex neural network model into real and imaginary parts, and a complex convolution kernel is designed in the complex convolution layer to simulate complex operations; assuming that the speech data input to the complex neural network model is M=X+iY, X and Y are the real and imaginary components of the speech data M respectively, and the complex convolution kernel vector is K=A+iB, A and B are the real and imaginary parts of the complex convolution kernel, then the speech data M is operated through the complex convolution layer, and the output is:

[0068] M*K=(X*AY*B)+i(X*B+Y*A)

[0069] Converted into a form that conforms to the complex convolution kernel operation, expressed as;

[0070]

[0071] Among them, * represents the convolution operation, (M*K) real 、(M*K) imag Respectively represent the real and imaginary components of M*K;

[0072] The activation function of the complex activation layer is the PEReLU function or the PCReLU function. For the speech data C input into the complex neural network model, the expression of the PEReLU function is:

[0073] PEReLU(C)=EC 3 +FC 2 +GC+H

[0074] The expression of the PCReLU function is:

[0075]

[0076] Among them, C real , C imag Respectively represent the real part and imaginary part of C, E, F, G, H are the polynomial coefficients of the PEReLU function, E1, F1, G1, H1 are the polynomial coefficients of the real part of the PEReLU function, and E2, F2, G2, H2 are the polynomial coefficients of the imaginary part of the PEReLU function; in this embodiment,

[0077] PEReLU(C)=3.5×10 -6 C 3 +0.188C 2+0.5C+0.234

[0078] The expression of the PCReLU function is:

[0079]

[0080] Among them, C real , C imag represent the real and imaginary parts of C respectively;

[0081] The fully connected network unit includes a first complex fully connected layer, an activation function layer and a second complex fully connected layer. The activation function layer is located between the first complex fully connected layer and the second complex fully connected layer. The activation function of the activation function layer is a PEReLU function or a PCReLU function. The expression of the PEReLU function is:

[0082] PEReLU(C)=3.5×10 -6 C 3 +0.188C 2 +0.5C+0.234

[0083] The expression of the PCReLU function is:

[0084]

[0085] Among them, C real , C imag Respectively represent the real part and imaginary part of C; the function here is obtained by fitting the objective function or its derivative by the least squares method for a given interval. The implementation method of the first complex fully connected layer and the second complex fully connected layer is the same as the complex convolutional layer, which is realized by splitting the complex vector and simulating complex operations. The output of the second complex fully connected layer is used as the output layer of the fully connected network unit. The adaptability of the constructed complex neural network model to the encrypted speech form and the design of the activation function facilitate the subsequent speech recognition execution server to efficiently and accurately recognize the encrypted speech in the encrypted domain.

[0086] When training the complex neural network model in step S2 of the present embodiment, the speech data of the training set is used as the input of the complex neural network model, and the label corresponding to the speech data of the training set is used as the output label of the complex neural network model. The loss function adopts cross entropy and is trained by the gradient descent method. The specific process is not repeated here. The trained complex neural network model is encapsulated in the speech recognition execution server for standby.

[0087] In step S4, the client uses the public key pk to perform data encoding and encryption on the speech to be recognized:

[0088] Assume that there are n voice data in the speech to be recognized, represented by: 0 , v1 , …, v n-1 , each voice data has m points: s=0,1,…,n-1,the formula is executed for all speech data:

[0089]

[0090] Wherein, j = 0, 1, ..., m-1;

[0091] Encode n voice data into m vectors V 0 , V 1 , …, V m-1 ,in:

[0092]

[0093] Use the public key pk to identify m vectors V 0 , V 1 , …, V m-1 Encrypt and get the encrypted data:

[0094] During the speech encryption and decryption stage, a privacy server is used to generate a public key and a private key and send them to the client, and the public key is sent to the speech recognition execution server. The client then encodes and encrypts the speech data and sends it to the speech recognition execution server. The speech recognition execution server uses a complex neural network model to recognize the encrypted speech in the encrypted domain. When the complex neural network model in the speech recognition execution server performs speech recognition on the encrypted speech in the encrypted domain, the multiplication used is the homomorphic multiplication of the encryption domain, and the addition used is the homomorphic addition of the encryption domain. The adaptability of the complex neural network model facilitates the speech recognition execution server to efficiently recognize the encrypted speech in the encrypted domain, and then sends the recognition result to the client, which decrypts it to obtain the speech recognition result.

[0095] Example 2

[0096] This embodiment specifically illustrates the convolutional neural network model proposed in Embodiment 1. The one-dimensional convolutional network unit includes six convolution blocks, each of which is composed of a complex convolution layer, a complex activation layer and an average pooling layer; wherein the activation function of the complex activation layer selects the PEReLU function, and the input speech data is a speech signal of a length of 1s sampled at 8000HZ, that is, the input dimension of the first complex convolution layer is (1,8000) speech data.

[0097] The convolution kernel size of the first complex convolution layer is 1×13, the number of convolution kernels is 32, the stride is 3, and the average pooling layer selects an average pooling layer with a stride and size of 3. The convolution kernel size of the second complex convolution layer is 1×11, the number of convolution kernels is 32, and the stride is 3. The average pooling layer selects an average pooling layer with a stride and size of 3. The convolution kernel size of the third complex convolution layer is 1×9, the number of convolution kernels is 64, the stride is 3, and the average pooling layer selects an average pooling layer with a stride and size of 3. The convolution kernel size of the fourth complex convolution layer is 1×7, the number of convolution kernels is 128, the stride is 3, and the average pooling layer selects an average pooling layer with a stride and size of 3. The convolution kernel size of the fifth complex convolution layer is 1×5, the number of convolution kernels is 128, the stride is 3, and the average pooling layer selects an average pooling layer with a stride and size of 3. The convolution kernel size of the sixth complex convolution layer is 1×3, the number of convolution kernels is 192, the stride is 3, and the average pooling layer selects an average pooling layer with a stride and size of 3.

[0098] The output result of the one-dimensional convolutional network unit is used as the input of the fully connected network unit. The fully connected network unit includes two complex fully connected layers and an activation function layer, wherein the activation function layer is located between the first complex fully connected layer and the second complex fully connected layer, and the PEReLU function is used. The second complex fully connected layer is used as the output layer of the network. The two fully connected layers contain 256 and 35 neurons respectively. The last fully connected layer is used as the output layer, and the output result is linearly mapped to 35 categories. Finally, the output modulus is taken as the final classification result.

[0099] Example 3

[0100] like Figure 3 As shown, the present application also proposes an encrypted domain speech recognition system based on a complex neural network, comprising:

[0101] The data set collection unit 1 is used to collect a speech data set and divide the speech data set into a training set and a test set, wherein the speech data in the training set and the test set are both regarded as plural;

[0102] The complex neural network construction training module 2 is used to construct a complex neural network model, and train the complex neural network model using a training set, and test the accuracy of the complex neural network model using a test set to obtain a final complex neural network model, which is located in the speech recognition execution server 3;

[0103] The privacy server 4 is used to generate a public key pk and a private key sk, and send both the public key pk and the private key sk to the client 5, and send the public key pk to the speech recognition execution server 3;

[0104] The client 5 uses the public key pk to encode and encrypt the speech to be recognized. Each speech to be recognized has a number of tags. Each voice data of the encrypted speech to be recognized is regarded as a plural number and sent to the speech recognition execution server 3.

[0105] The speech recognition execution server 3 is provided with a complex neural network construction training module 2, which performs speech recognition on the encrypted speech in the encryption domain, obtains the encryption probabilities of several labels corresponding to the encrypted speech, and sends them to the client 5;

[0106] The client 5 also uses the private key sk to decrypt the encrypted probabilities of several labels, and takes the label with the largest probability value as the result of speech recognition.

[0107] See also Figure 3 The system also includes: a sampling processing module 6. After the data set acquisition unit 1 acquires the speech data set, the sampling processing module 6 samples the speech data in the speech data set to make it speech data with a sampling frequency of 8000 Hz and a length of 1 second, matching the input dimension of the complex neural network model.

[0108] Obviously, the above embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for speech recognition in encrypted domain based on complex neural network, characterized in that: include: S1. Collect a speech data set, divide the speech data set into a training set and a test set, and treat the speech data in the training set and the test set as plural; S2. Construct a complex neural network model, train the complex neural network model using the training set, test the accuracy of the complex neural network model using the test set, obtain the final complex neural network model, and encapsulate it in the speech recognition execution server; The complex neural network model constructed in step S2 includes a one-dimensional convolutional network unit and a fully connected network unit. The speech data is output as a one-dimensional feature vector through the one-dimensional convolutional network unit, and the one-dimensional feature vector is used as the input of the fully connected network unit; the one-dimensional convolutional network unit includes six convolution blocks, each of which is composed of a complex convolution layer, a complex activation layer and an average pooling layer. The complex convolution layer splits the speech data input to the complex neural network model into a real part and an imaginary part, and a complex convolution kernel is designed in the complex convolution layer to simulate complex operations; assuming that the speech data input to the complex neural network model is M=X+iY, X and Y are the real and imaginary components of the speech data M respectively, and the complex convolution kernel vector is K=A+iB, A and B are the real and imaginary parts of the complex convolution kernel, then the speech data M is operated through the complex convolution layer, and the output is: M*K=(X*AY*B)+i(X*B+Y*A) Converted into a form that conforms to the complex convolution kernel operation, expressed as; Among them, * represents the convolution operation, (M*K) real 、(M*K) imag Respectively represent the real and imaginary components of M*K; The activation function of the complex activation layer is the PEReLU function or the PCReLU function. For the speech data C input into the complex neural network model, the expression of the PEReLU function is: PERELU(C)=EC 3 +FC 2 +HC+H The expression of the PCReLU function is: Among them, C real , C imag Represent the real and imaginary parts of C respectively, E, F, G, H are the polynomial coefficients of the PEReLU function, E1, F1, G1, H1 are the polynomial coefficients of the real part of the PEReLU function, E2, F2, G2, H2 are the polynomial coefficients of the imaginary part of the PEReLU function; The fully connected network unit includes a first complex fully connected layer, an activation function layer and a second complex fully connected layer, the activation function layer is located between the first complex fully connected layer and the second complex fully connected layer, the activation function of the activation function layer is a PEReLU function or a PCReLU function, the first complex fully connected layer and the second complex fully connected layer are both implemented by splitting complex vectors and simulating complex operations, and the output of the second complex fully connected layer is used as the output layer of the fully connected network unit; S3. Generate a public key pk and a private key sk using a privacy server, send both the public key pk and the private key sk to the client, and send the public key pk to the speech recognition execution server; S4. The client uses the public key pk to encode and encrypt the voice to be recognized. Each voice to be recognized has several tags. Each voice data of the encrypted voice to be recognized is regarded as a plural number and sent to the voice recognition execution server; S5. The plural neural network model in the speech recognition execution server performs speech recognition on the encrypted speech in the encryption domain, and obtains the encryption probability of several labels corresponding to the encrypted speech; S6. The speech recognition execution server sends the encrypted probabilities of several tags to the client. After the client decrypts the tags using the private key sk, it takes the tag with the largest probability value as the result of speech recognition.

2. The encrypted domain speech recognition method based on a complex neural network according to claim 1, characterized in that: In step S1, each keyword in the speech data set is composed of a set of Z seconds of speech data from different speech sources. The speech data is provided with labels, and each piece of speech data is labeled with one of several keywords.

3. The encrypted domain speech recognition method based on a complex neural network according to claim 2, characterized in that: After the speech data set is collected, the speech data in the speech data set is sampled to become speech data with a sampling frequency of Q and a length of Z seconds.

4. The encrypted domain speech recognition method based on a complex neural network according to claim 1, characterized in that: When training the complex neural network model, the speech data of the training set is used as the input of the complex neural network model, and the labels corresponding to the speech data of the training set are used as the output labels of the complex neural network model. The loss function adopts cross entropy and is trained by gradient descent method.

5. The encrypted domain speech recognition method based on complex neural network according to claim 4 is characterized in that: In step S4, the client uses the public key pk to perform data encoding and encryption on the speech to be recognized: Assume that there are n voice data in the speech to be recognized, represented by: 0 ,v 1 ,…,v n-1 , each voice data has m points: Execute the formula for all voice data: Where, j = 0, 1, ..., m-1; Encode n voice data into m vectors V 0 ,V 1 ,…,V m-1 ,in: Use the public key pk to identify m vectors V 0 ,V 1 ,…,V m-1 Encrypt and get the encrypted data:

6. The method for encrypted domain speech recognition based on complex neural network according to claim 5, characterized in that: In step S5, when the complex neural network model in the speech recognition execution server performs speech recognition on the encrypted speech in the encryption domain, the multiplication used is the homomorphic multiplication of the encryption domain.

7. The encrypted domain speech recognition method based on a complex neural network according to claim 5, characterized in that: In step S5, when the complex neural network model in the speech recognition execution server performs speech recognition on the encrypted speech in the encryption domain, the addition used is the homomorphic addition of the encryption domain.

8. An encrypted domain speech recognition system based on a complex neural network, characterized in that: include: A data set collection unit, used for collecting a speech data set and dividing the speech data set into a training set and a test set, wherein the speech data in the training set and the test set are both regarded as plural; A complex neural network construction training module is used to construct a complex neural network model, train the complex neural network model using a training set, test the accuracy of the complex neural network model using a test set, and obtain a final complex neural network model, which is located in the speech recognition execution server; The constructed complex neural network model includes a one-dimensional convolutional network unit and a fully connected network unit. The speech data is output as a one-dimensional feature vector through the one-dimensional convolutional network unit, and the one-dimensional feature vector is used as the input of the fully connected network unit; the one-dimensional convolutional network unit includes six convolution blocks, each of which is composed of a complex convolution layer, a complex activation layer and an average pooling layer. The complex convolution layer splits the speech data input to the complex neural network model into a real part and an imaginary part, and a complex convolution kernel is designed in the complex convolution layer to simulate complex operations; assuming that the speech data input to the complex neural network model is M=X+iY, X and Y are the real part and imaginary part of the speech data M respectively, and the complex convolution kernel vector is K=A+iB, A and B are the real part and imaginary part of the complex convolution kernel, then the speech data M is operated through the complex convolution layer, and the output is: M*K=(X*AY*B)+i(X*B+Y*A) Converted into a form that conforms to the complex convolution kernel operation, expressed as; Among them, * represents the convolution operation, (M*K) real 、(M*K) imag Respectively represent the real and imaginary components of M*K; The activation function of the complex activation layer is the PEReLU function or the PCReLU function. For the speech data C input into the complex neural network model, the expression of the PEReLU function is: PEReLU(C)=FC 3 +FC 2 +GC+H The expression of the PCReLU function is: Among them, C real , C imag Represent the real and imaginary parts of C respectively, E, F, G, H are the polynomial coefficients of the PEReLU function, E1, F1, G1, H1 are the polynomial coefficients of the real part of the PEReLU function, E2, F2, G2, H2 are the polynomial coefficients of the imaginary part of the PEReLU function; The fully connected network unit includes a first complex fully connected layer, an activation function layer and a second complex fully connected layer, the activation function layer is located between the first complex fully connected layer and the second complex fully connected layer, the activation function of the activation function layer is a PEReLU function or a PCReLU function, the first complex fully connected layer and the second complex fully connected layer are both implemented by splitting complex vectors and simulating complex operations, and the output of the second complex fully connected layer is used as the output layer of the fully connected network unit; The privacy server is used to generate a public key pk and a private key sk, and send both the public key pk and the private key sk to the client, and send the public key pk to the speech recognition execution server; The client uses the public key pk to encode and encrypt the speech to be recognized. Each speech to be recognized has several tags. Each voice data of the encrypted speech to be recognized is regarded as a plural number and sent to the speech recognition execution server. The speech recognition execution server has a complex neural network construction training module, which performs speech recognition on the encrypted speech in the encrypted domain, obtains the encryption probabilities of several labels corresponding to the encrypted speech, and sends them to the client; The client also uses the private key sk to decrypt the encrypted probabilities of several labels, and takes the label with the largest probability value as the result of speech recognition.

9. The encrypted domain speech recognition system based on complex neural network according to claim 8, characterized in that: Also includes: The sampling processing module samples the speech data in the speech data set after the data set acquisition unit acquires the speech data set, so that it becomes speech data with a sampling frequency of Q and a length of Z seconds, matching the input dimension of the complex neural network model.

Images