Voice data processing method and device based on privacy protection and readable storage medium

Abstract

The invention relates to the technical field of digital medical treatment, and provides a voice data processing method and device based on privacy protection, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining a to-be-processed voice signal; preprocessing the voice signal to be processed to obtain first voice data; determining private data from the first voice data; creating corresponding confusion sample data according to the privacy data; combining the confused sample data with the first voice data to obtain combined voice data; and performing recognition processing on the combined voice data based on a pre-trained acoustic federal learning model to obtain a voice recognition result. According to the technical scheme, the voice recognition process of the digital medical industry is safer, and the privacy of the user is not easy to leak.

Description

Technical Field

[0001] The embodiments of the present application relate to, but are not limited to, the field of digital medical technology, and in particular, to a voice data processing method, device, electronic device, and computer-readable storage medium based on privacy protection. Background Art

[0002] With the continuous development of science and technology, digital medicine has been continuously promoted and applied in life; intelligent medical consultation has become more and more common in the field of digital medicine. Users can consult conversational robots about simple medical issues or conversational robots can promote health knowledge and current popular medical information to users; however, in the process of intelligent medical consultation, due to the defects of existing voice recognition models, it is easy for criminals to understand the user's privacy data through relevant background data, which may cause the user's privacy data to be leaked, bringing unsafe factors to users. Summary of the invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] In order to solve the problems mentioned in the above background technology, the embodiments of the present application provide a voice data processing method, device, electronic device and computer-readable storage medium based on privacy protection, so that the voice recognition process in the digital medical industry is safer and the user's privacy will not be easily leaked.

[0005] In a first aspect, an embodiment of the present application provides a voice data processing method based on privacy protection, comprising:

[0006] Obtaining a voice signal to be processed;

[0007] Preprocessing the speech signal to be processed to obtain first speech data;

[0008] Determining private data from the first voice data; and creating corresponding obfuscated sample data according to the private data;

[0009] Combining the obfuscated sample data with the first voice data to obtain combined voice data;

[0010] The combined speech data is recognized and processed based on a pre-trained acoustic federated learning model to obtain a speech recognition result.

[0011] According to some embodiments of the present application, the preprocessing of the speech signal to be processed to obtain the first speech data includes:

[0012] Performing pre-emphasis processing on the speech signal to be processed to obtain a first speech signal;

[0013] Performing filtering on the first speech signal to obtain a second speech signal;

[0014] The second speech signal is subjected to windowing and framing processing to obtain the first speech data.

[0015] According to some embodiments of the present application, determining the privacy data from the first voice data includes:

[0016] Performing text conversion processing on the first voice data to obtain voice data text;

[0017] Performing keyword screening processing on the voice data text to obtain a keyword text;

[0018] The first voice data corresponding to the keyword text is determined as the privacy data.

[0019] According to some embodiments of the present application, the creating corresponding obfuscated sample data according to the private data includes:

[0020] Performing gradient calculation on the privacy data to obtain a privacy gradient value;

[0021] Adjusting the privacy gradient value according to a preset gradient adjustment strategy to obtain an adjusted gradient value;

[0022] The corresponding obfuscated sample data is generated according to the preset inner product constraint condition and the adjusted gradient value.

[0023] According to some embodiments of the present application, the acoustic federated learning model includes a convolutional layer module, a pooling layer module, and a fully connected layer module, and the pre-trained acoustic federated learning model performs recognition processing on the combined speech data to obtain a speech recognition result, including:

[0024] Performing feature extraction processing on the combined speech data based on the convolutional layer module to obtain combined speech features;

[0025] Based on the pooling layer module, average pooling processing is performed on the combined speech features to obtain pooled speech features;

[0026] Based on the fully connected layer module, the pooled speech features are classified and processed to obtain a preliminary speech recognition result;

[0027] The speech recognition result is obtained by correcting the preliminary speech recognition result according to the confusing sample data.

[0028] According to some embodiments of the present application, the step of performing correction processing on the preliminary speech recognition result according to the confused sample data to obtain the speech recognition result includes:

[0029] Analyzing and processing the obfuscated sample data to obtain an obfuscated sample label;

[0030] The speech recognition result is obtained by performing information elimination processing on the preliminary speech recognition result according to the confusion sample mark.

[0031] According to some embodiments of the present application, after the pre-trained acoustic federated learning model performs recognition processing on the combined speech data to obtain a speech recognition result, the method further includes:

[0032] Converting the speech recognition result into encrypted text;

[0033] The encrypted text is sent to a preset network address.

[0034] In a second aspect, an embodiment of the present application further provides a voice data processing device based on privacy protection, the device comprising:

[0035] A first processing module, used for acquiring a speech signal to be processed;

[0036] A second processing module, used for preprocessing the speech signal to be processed to obtain first speech data;

[0037] A third processing module, configured to determine private data from the first voice data; and to create corresponding obfuscated sample data according to the private data;

[0038] A fourth processing module, used for combining the obfuscated sample data with the first voice data to obtain combined voice data;

[0039] The fifth processing module is used to perform recognition processing on the combined speech data based on a pre-trained acoustic federated learning model to obtain a speech recognition result.

[0040] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the voice data processing method based on privacy protection as described in the first aspect above is implemented.

[0041] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the voice data processing method based on privacy protection as described in the first aspect above.

[0042] According to the privacy-protected voice data processing method of the embodiment provided by the present application, at least the following beneficial effects are achieved: in the process of voice data processing based on privacy protection, firstly, the voice signal to be processed is obtained; then, the voice signal to be processed is preprocessed to obtain the first voice data; then, the privacy data is determined from the first voice data; and corresponding obfuscated sample data is created according to the privacy data; then, the obfuscated sample data is combined with the first voice data to obtain the combined voice data; finally, the combined voice data obtained in the above steps is recognized and processed based on the pre-trained acoustic federated learning model to obtain the voice recognition result. Through the above technical solution, the voice recognition process in the digital medical industry is made safer, and the user's privacy will not be easily leaked. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0044] Figure 1 is a flow chart of a voice data processing method based on privacy protection provided by an embodiment of the present application;

[0045] Figure 2 This is a flowchart of preprocessing a speech signal to be processed in a voice data processing method based on privacy protection provided by an embodiment of the present application;

[0046] Figure 3 It is a flowchart of determining private data from first voice data in a voice data processing method based on privacy protection provided by an embodiment of the present application;

[0047] Figure 4 It is a flowchart of creating obfuscated sample data based on privacy data in a voice data processing method based on privacy protection provided by an embodiment of the present application;

[0048] Figure 5 This is a flow chart of identifying and processing combined voice data in a voice data processing method based on privacy protection provided by an embodiment of the present application;

[0049] Figure 6 It is a flowchart of correcting the initial speech recognition result in a voice data processing method based on privacy protection provided by an embodiment of the present application;

[0050] Figure 7 is a flowchart of a voice data processing method based on privacy protection provided by another embodiment of the present application;

[0051] Figure 8is a schematic diagram of a voice data processing device based on privacy protection provided by an embodiment of the present application;

[0052] Fig. 9 It is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0055] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0056] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0057] AI is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Artificial intelligence is a branch of computer science. Artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0058] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0059] Artificial intelligence is AI. AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0060] The servers involved in artificial intelligence technology can be independent servers or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0061] The present application provides a method, device, electronic device and computer-readable storage medium for voice data processing based on privacy protection. In the process of voice data processing based on privacy protection, firstly, a voice signal to be processed is obtained; then, the voice signal to be processed is preprocessed to obtain the first voice data; then, the privacy data is determined from the first voice data; and corresponding obfuscated sample data is created according to the privacy data; then, the obfuscated sample data is combined with the first voice data to obtain the combined voice data; finally, the combined voice data obtained in the above steps is recognized and processed based on the pre-trained acoustic federated learning model to obtain the voice recognition result. Through the above technical solution, the voice recognition process in the digital medical industry is made safer, and the user's privacy will not be easily leaked.

[0062] The voice data processing method based on privacy protection provided in the embodiment of the present application relates to the field of digital medical technology. The voice data processing method based on privacy protection provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements a voice data processing method based on privacy protection, etc., but is not limited to the above forms.

[0063] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0064] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0065] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0066] like Figure 1 As shown, Figure 1It is a flowchart of a voice data processing method based on privacy protection provided by an embodiment of the present application. The voice data processing method based on privacy protection includes but is not limited to steps S100 to S500.

[0067] Step S100, obtaining a speech signal to be processed;

[0068] Step S200, preprocessing the speech signal to be processed to obtain first speech data;

[0069] Step S300, determining private data from the first voice data; and creating corresponding obfuscated sample data according to the private data;

[0070] Step S400, combining the obfuscated sample data with the first voice data to obtain combined voice data;

[0071] Step S500: Perform recognition processing on the combined speech data based on the pre-trained acoustic federated learning model to obtain a speech recognition result.

[0072] It should be noted that in the process of voice data processing based on privacy protection, firstly, the voice signal to be processed is obtained; then the voice signal to be processed is preprocessed to obtain the first voice data; then the privacy data is determined from the first voice data; and the corresponding obfuscated sample data is created according to the privacy data; then the obfuscated sample data is combined with the first voice data to obtain the combined voice data; finally, the combined voice data obtained in the above steps is recognized and processed based on the pre-trained acoustic federated learning model to obtain the voice recognition result. Through the above technical solution, the voice recognition process in the digital medical industry is made safer and the user's privacy will not be easily leaked.

[0073] It should be noted that the voice signal to be processed is the voice signal generated by the user in the digital medical industry, for example, in the process of intelligent medical consultation. By way of example, the voice signal to be processed can be the voice signal generated by the user in the process of intelligent medical consultation using a conversational robot. The conversational robot can receive the voice signal generated by the user and perform recognition and response processing on the received voice signal.

[0074] It is worth noting that the first voice data can be obtained by preprocessing the voice signal to be processed, so as to prepare for the subsequent voice recognition processing. For example, during the intelligent consultation process, the dialogue robot can convert the received voice signal into the first voice data so that the dialogue robot can perform response processing.

[0075] It is understandable that the privacy data is determined from the first voice data, and the corresponding obfuscated sample data is created for the privacy data, and the privacy of the voice processing is increased based on the obfuscated sample data, which improves the confidentiality of the voice processing process and brings a better user experience to the user. The obfuscated sample data and the first voice data are combined and processed to obtain the combined voice data, and then when the criminals steal the combined voice data, they will not know which parts are the privacy data and which parts are the first voice data generated by the user, which improves the confidentiality of data transmission and ensures the security of data transmission.

[0076] It should be noted that the pre-trained acoustic federated learning model is an acoustic model that has been trained through federated learning. The trained acoustic model can recognize and process speech data. In the federated learning process, multiple organizations provide data samples to train the acoustic model, and each organization only provides its neural network gradient instead of the data itself. After the training is completed, the acoustic federated learning model can accurately recognize and process speech data, which greatly facilitates users in speech recognition.

[0077] It should be noted that federated learning is widely used in learning models from multiple data sources. Usually, the federated learning scheme consists of a central server that aggregates neural network gradients from multiple sources. Each organization contributes neural network gradients instead of the data itself and receives updates to the local model. Malicious behaviors such as membership inference and model inversion can still extract sensitive information from the secret gradient. In particular, when federated learning is applied to the field of speech recognition, the risk of sensitive information loss is greater when speech data is attacked. First, the speech synthesizer can construct an individual's voice, causing it to lose sole control of its own voice. Second, the voice cannot be reset like a password to ensure security. Finally, high-quality speech models need to be trained on a large amount of speech data, so model inversion attacks are more likely to expose sensitive information. Generally speaking, the server releases the privacy dataset by performing feature transformation, but in the federated context, the member inference attacker can access the feature transformation program, so feature transformation cannot provide any protection against such attacks. In addition, most feature transformations lack mathematical guarantees to ensure that the transformation is a one-way function, so it cannot be guaranteed that the privacy dataset cannot be obtained by reverse reconstruction.

[0078] In some embodiments, Figure 2 As shown, the above step S200 may include but is not limited to steps S210 to S230.

[0079] Step S210, performing pre-emphasis processing on the speech signal to be processed to obtain a first speech signal;

[0080] Step S220, filtering the first speech signal to obtain a second speech signal;

[0081] Step S230: performing windowing and framing processing on the second speech signal to obtain first speech data.

[0082] It should be noted that, in the process of preprocessing the voice signal to be processed, the voice signal to be processed is first pre-emphasized to obtain the first voice signal; then the first voice signal is filtered to obtain the second voice signal; finally, the second voice signal is windowed and framed to obtain the first voice data.

[0083] It is worth noting that the first voice signal can be obtained by pre-emphasis processing on the voice signal to be processed; pre-emphasis can compensate for the loss of high-frequency components in the voice signal and protect the channel information; the medium is the carrier of sound energy. Under the condition of a certain sound source size, the higher the frequency, the more serious the loss of sound energy by the medium. The second voice signal can be obtained by filtering the first voice signal. The filtering process can filter out the interference signal in the first voice signal, so that the subsequent voice recognition process can be more accurate. Framing is to divide the original voice signal into N segments of fixed size. Here, each segment of the voice signal is called a frame, and the frame length is generally 10 to 30ms; framing generally adopts the overlapping segmentation method to make the transition between frames smooth and maintain its continuity; the overlapping part of the previous frame and the next frame is called frame shift, and the ratio of frame shift to frame length is generally 0-0.5. After the signal is framed, in order to make a smooth transition between frames and maintain continuity, that is, to eliminate the signal discontinuity that may be caused at both ends of each frame, the signal needs to be truncated and framed and windowed. Because truncation will cause frequency domain energy leakage, the window function can reduce the impact of truncation and bring each frame into the window function to form a windowed speech signal.

[0084] In some embodiments, Figure 3 As shown, the above step S300 may also include but is not limited to steps S310 to S330.

[0085] Step S310, converting the first voice data into text to obtain voice data text;

[0086] Step S320, performing keyword screening processing on the voice data text to obtain a keyword text;

[0087] Step S330: determine the first voice data corresponding to the keyword text as private data.

[0088] It should be noted that in the process of determining the private data from the first voice data, the first voice data is first converted into text to obtain the voice data text; then the voice data text is filtered by keywords to obtain the keyword text; finally, the first voice data corresponding to the keyword text is determined as the private data. It can be understood that the private data is the data information that the user does not want other people to know. For example, the private data may include the user's name, the user's age, and the user's relevant physical condition.

[0089] Specifically, during the intelligent consultation process, the conversational robot can receive the voice signal to be processed generated by the user, and then pre-process the voice signal to be processed to obtain the first voice data; then perform text conversion processing on the first voice data to obtain the voice data text; then perform keyword screening processing on the voice data text to obtain the keyword text; finally, the first voice data corresponding to the keyword text is determined as private data. Among them, keyword screening can pre-set relevant keywords, such as the user's gender, name, age or the user's relevant physical condition can be used as private data, which is not limited here.

[0090] In some embodiments, Figure 4 As shown, the above step S300 may also include but is not limited to steps S340 to S360.

[0091] Step S340, performing gradient calculation processing on the privacy data to obtain a privacy gradient value;

[0092] Step S350, adjusting the privacy gradient value according to a preset gradient adjustment strategy to obtain an adjusted gradient value;

[0093] Step S360: Generate corresponding obfuscated sample data according to the preset inner product constraint condition and the adjusted gradient value.

[0094] It should be noted that in the process of creating obfuscated sample data, the privacy gradient value can be obtained by first performing gradient calculation on the private data; then the privacy gradient value can be adjusted according to the preset gradient adjustment strategy to obtain the adjusted gradient value; finally, the corresponding obfuscated sample data can be created and generated according to the preset inner product constraint adjustment and adjustment gradient value.

[0095] It can be understood that the privacy gradient value can be obtained by performing gradient calculation on the privacy data; then the privacy gradient value can be adjusted and corrected according to the preset gradient adjustment strategy to obtain the adjusted gradient value; finally, the corresponding obfuscated sample data can be generated according to the preset inner product constraint and the adjusted gradient value, so as to prepare for the subsequent speech privacy recognition. The privacy gradient value is adjusted according to the preset gradient adjustment strategy. For example, the original privacy gradient value is calculated to be 0.8, but the preset gradient adjustment strategy is that the privacy gradient value is at least 0.9, so that the original privacy gradient value is corrected from 0.8 to 0.9 according to the preset gradient adjustment strategy; finally, the corresponding obfuscated sample data can be generated according to the preset inner product constraint (for example, the inner product constraint is that the inner product is greater than or equal to 0) and the corrected privacy gradient value 0.9. Among them, the corresponding obfuscated sample data can be obtained by inputting the preset inner product constraint and the adjusted gradient value into the predefined objective function. Among them, in order to ensure that the attacker can be confused, the obfuscated sample and the privacy data sample have a high similarity in gradient; in order to protect the privacy data, the obfuscated sample and the privacy data sample are not visually similar.

[0096] In some embodiments, Figure 5 As shown, the acoustic federated learning model includes a convolutional layer module, a pooling layer module and a fully connected layer module, and the above step S500 may also include but is not limited to steps S510 to S540.

[0097] Step S510, performing feature extraction processing on the combined speech data based on the convolutional layer module to obtain combined speech features;

[0098] Step S520, performing average pooling processing on the combined speech features based on the pooling layer module to obtain pooled speech features;

[0099] Step S530, classifying and processing the pooled speech features based on the fully connected layer module to obtain a preliminary speech recognition result;

[0100] Step S540, correcting the preliminary speech recognition result according to the obfuscated sample data to obtain a speech recognition result.

[0101] It should be noted that in the process of recognizing and processing the combined speech data, first, the combined speech features are extracted based on the convolutional layer module to obtain the combined speech features; then, the combined speech features are averaged pooled based on the pooling layer to obtain the pooled speech features; then, the pooled speech features are classified based on the fully connected layer module to obtain the preliminary speech recognition results; finally, the preliminary speech recognition results are corrected based on the confused sample data to obtain the speech recognition results.

[0102] For example, in the digital medical industry, during the intelligent consultation process, the user needs to have a conversation with the dialogue robot. The combined voice features can be obtained by extracting the features of the combined voice data based on the convolutional layer module. Then, the combined voice features can be averaged and pooled based on the pooling layer module to obtain the pooled voice features. Then, the pooled voice features can be classified and processed based on the fully connected layer module to obtain the preliminary voice recognition results. Finally, the preliminary voice recognition results can be corrected according to the confused sample data to obtain the voice recognition results. Among them, the pre-trained acoustic federated learning model is a model trained in advance by means of federated learning, and the relevant parameters in the model have been corrected and adjusted. The preliminary voice recognition results can be corrected according to the confused sample data to eliminate the confused sample data in the preliminary voice recognition results.

[0103] In some embodiments, Figure 6 As shown, the above step S540 may also include but is not limited to steps S541 to S542.

[0104] Step S541, analyzing and processing the obfuscated sample data to obtain an obfuscated sample label;

[0105] Step S542: performing information elimination processing on the preliminary speech recognition result according to the confusion sample mark to obtain the speech recognition result.

[0106] It should be noted that the confusion sample data can be analyzed and processed to obtain the confusion sample label; then the initial semantic recognition result is subjected to information elimination processing according to the confusion sample label to obtain the speech recognition result. For example, in the process of intelligent medical consultation, the confusion sample data can be analyzed and processed to obtain the corresponding confusion sample label; then the initial speech recognition result is subjected to information elimination processing according to the confusion sample label to obtain the corresponding speech recognition result.

[0107] It is understandable that the confused sample data can be analyzed and processed to obtain the confused sample tags corresponding to the confused sample data; then the initial speech recognition results can be processed to remove information based on the confused sample tags to obtain the complete speech recognition results. In the process of intelligent medical consultation based on the dialogue robot, the confused sample data can be analyzed and processed to obtain the confused sample tags; then the initial speech recognition results can be processed to remove the confused sample data in the initial speech recognition results based on the confused sample tags, and then the corresponding speech recognition results can be obtained.

[0108] In some embodiments, Figure 7 As shown, after executing the above step S500, steps S610 to S620 may also be included but not limited to.

[0109] Step S610, converting the speech recognition result into encrypted text;

[0110] Step S620, sending the encrypted text to a preset network address.

[0111] It should be noted that after obtaining the speech recognition result, the speech recognition result can also be converted into encrypted text; and then the encrypted text is sent to a pre-set network address.

[0112] For example, in the process of intelligent consultation, after obtaining the voice recognition result, the voice recognition result can also be converted into encrypted text; then the obtained encrypted text is sent to the network address reserved by the user to facilitate the user's subsequent viewing operation. Among them, converting the voice recognition result into encrypted text can further protect the user's privacy and improve security.

[0113] In addition, if Figure 8 As shown, an embodiment of the present application further provides a voice data processing device 10 based on privacy protection, comprising:

[0114] The first processing module 100 is used to obtain a speech signal to be processed;

[0115] The second processing module 200 is used to pre-process the speech signal to be processed to obtain the first speech data;

[0116] The third processing module 300 is used to determine the private data from the first voice data; and to create corresponding obfuscated sample data according to the private data;

[0117] A fourth processing module 400, configured to combine the obfuscated sample data with the first voice data to obtain combined voice data;

[0118] The fifth processing module 500 is used to perform recognition processing on the combined speech data based on the pre-trained acoustic federated learning model to obtain a speech recognition result.

[0119] It should be noted that in the process of voice data processing based on privacy protection, firstly, the voice signal to be processed is obtained; then the voice signal to be processed is preprocessed to obtain the first voice data; then the privacy data is determined from the first voice data; and the corresponding obfuscated sample data is created according to the privacy data; then the obfuscated sample data is combined with the first voice data to obtain the combined voice data; finally, the combined voice data obtained in the above steps is recognized and processed based on the pre-trained acoustic federated learning model to obtain the voice recognition result. Through the above technical solution, the voice recognition process in the digital medical industry is made safer and the user's privacy will not be easily leaked.

[0120] It should be noted that the voice signal to be processed is the voice signal generated by the user in the digital medical industry, for example, in the process of intelligent medical consultation. By way of example, the voice signal to be processed can be the voice signal generated by the user in the process of intelligent medical consultation using a conversational robot. The conversational robot can receive the voice signal generated by the user and perform recognition and response processing on the received voice signal.

[0121] It is worth noting that the first voice data can be obtained by preprocessing the voice signal to be processed, so as to prepare for the subsequent voice recognition processing. For example, during the intelligent consultation process, the dialogue robot can convert the received voice signal into the first voice data so that the dialogue robot can perform response processing.

[0122] It is understandable that the privacy data is determined from the first voice data, and the corresponding obfuscated sample data is created for the privacy data, and the privacy of the voice processing is increased based on the obfuscated sample data, which improves the confidentiality of the voice processing process and brings a better user experience to the user. The obfuscated sample data and the first voice data are combined and processed to obtain the combined voice data, and then when the criminals steal the combined voice data, they will not know which parts are the privacy data and which parts are the first voice data generated by the user, which improves the confidentiality of data transmission and ensures the security of data transmission.

[0123] It should be noted that the pre-trained acoustic federated learning model is an acoustic model that has been trained through federated learning. The trained acoustic model can recognize and process speech data. In the federated learning process, multiple organizations provide data samples to train the acoustic model, and each organization only provides its neural network gradient instead of the data itself. After the training is completed, the acoustic federated learning model can accurately recognize and process speech data, which greatly facilitates users in speech recognition.

[0124] It should be noted that federated learning is widely used in learning models from multiple data sources. Usually, the federated learning scheme consists of a central server that aggregates neural network gradients from multiple sources. Each organization contributes neural network gradients instead of the data itself and receives updates to the local model. Malicious behaviors such as membership inference and model inversion can still extract sensitive information from the secret gradient. In particular, when federated learning is applied to the field of speech recognition, the risk of sensitive information loss is greater when speech data is attacked. First, the speech synthesizer can construct an individual's voice, causing it to lose sole control of its own voice. Second, the voice cannot be reset like a password to ensure security. Finally, high-quality speech models need to be trained on a large amount of speech data, so model inversion attacks are more likely to expose sensitive information. Generally speaking, the server releases the privacy dataset by performing feature transformation, but in the federated context, the member inference attacker can access the feature transformation program, so feature transformation cannot provide any protection against such attacks. In addition, most feature transformations lack mathematical guarantees to ensure that the transformation is a one-way function, so it cannot be guaranteed that the privacy dataset cannot be obtained by reverse reconstruction.

[0125] The specific implementation of the voice data processing device 10 based on privacy protection is basically the same as the specific implementation of the voice data processing method based on privacy protection described above, and will not be repeated here.

[0126] In addition, if Fig. 9 As shown, an embodiment of the present application further provides an electronic device 700 , which includes: a memory 720 , a processor 710 , and a computer program stored in the memory 720 and executable on the processor 710 .

[0127] The processor 710 and the memory 720 may be connected via a bus or in other ways.

[0128] The non-transient software program and instructions required to implement the privacy protection-based voice data processing method of the above-mentioned embodiments are stored in the memory 720. When executed by the processor 710, the privacy protection-based voice data processing method of the above-mentioned embodiments is executed, for example, the above-described Figure 1 The method steps S100 to S500, Figure 2 Steps S210 to S230 of the method, Figure 3 Steps S310 to S330 of the method, Figure 4 Steps S340 to S360 of the method, Figure 5 Steps S510 to S540 of the method, Figure 6 Steps S541 to S542 of the method and Figure 7 Method steps S610 to S620 in .

[0129] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0130] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions, which are executed by a processor 710 or a controller, for example, by a processor 710 in the above-mentioned device embodiment, so that the processor 710 can execute the voice data processing method based on privacy protection in the above-mentioned embodiment, for example, execute the above-mentioned Figure 1 Steps S100 to S500 of the method, Figure 2 Steps S210 to S230 of the method, Figure 3 Steps S310 to S330 of the method, Figure 4 Steps S340 to S360 of the method, Figure 5 Steps S510 to S540 of the method, Figure 6 Steps S541 to S542 of the method and Figure 7 Method steps S610 to S620 in.

[0131] The above embodiments may be used in combination, and modules with the same name in different embodiments may be the same or different.

[0132] The above describes specific embodiments of the present application, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0133] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0134] The apparatus, device, computer-readable storage medium and method provided in the embodiments of the present application correspond to each other. Therefore, the apparatus, device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding apparatus, device and computer storage medium will not be repeated here.

[0135] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0136] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0137] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0138] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0139] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0140] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0143] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0144] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (Flash RAM). The memory is an example of a computer-readable medium.

[0145] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0146] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0147] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0148] Embodiments of the present application may be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Embodiments of the present application may also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0149] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0150] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A voice data processing method based on privacy protection, characterized in that: include: Obtaining a voice signal to be processed; Preprocessing the speech signal to be processed to obtain first speech data; determining private data from the first voice data; and creating corresponding obfuscated sample data according to the private data; Combining the obfuscated sample data with the first voice data to obtain combined voice data; The combined speech data is recognized and processed based on a pre-trained acoustic federated learning model to obtain a speech recognition result.

2. The voice data processing method based on privacy protection according to claim 1 is characterized in that: The preprocessing of the speech signal to be processed to obtain the first speech data includes: Performing pre-emphasis processing on the speech signal to be processed to obtain a first speech signal; Performing filtering on the first speech signal to obtain a second speech signal; The second speech signal is subjected to windowing and framing processing to obtain the first speech data.

3. The voice data processing method based on privacy protection according to claim 1 is characterized in that: The determining of the privacy data from the first voice data includes: Performing text conversion processing on the first voice data to obtain voice data text; Performing keyword screening processing on the voice data text to obtain a keyword text; The first voice data corresponding to the keyword text is determined as the privacy data.

4. The voice data processing method based on privacy protection according to claim 1 is characterized in that: The step of creating corresponding obfuscated sample data according to the private data includes: Performing gradient calculation on the privacy data to obtain a privacy gradient value; Adjusting the privacy gradient value according to a preset gradient adjustment strategy to obtain an adjusted gradient value; The corresponding obfuscated sample data is generated according to the preset inner product constraint condition and the adjusted gradient value.

5. The voice data processing method based on privacy protection according to claim 1 is characterized in that: The acoustic federated learning model includes a convolutional layer module, a pooling layer module and a fully connected layer module. The pre-trained acoustic federated learning model performs recognition processing on the combined speech data to obtain a speech recognition result, including: Performing feature extraction processing on the combined speech data based on the convolutional layer module to obtain combined speech features; Based on the pooling layer module, average pooling processing is performed on the combined speech features to obtain pooled speech features; Based on the fully connected layer module, the pooled speech features are classified and processed to obtain a preliminary speech recognition result; The speech recognition result is obtained by correcting the preliminary speech recognition result according to the confusing sample data.

6. The voice data processing method based on privacy protection according to claim 5 is characterized in that: The correcting the preliminary speech recognition result according to the obfuscated sample data to obtain the speech recognition result includes: Analyzing and processing the obfuscated sample data to obtain an obfuscated sample label; The speech recognition result is obtained by performing information elimination processing on the preliminary speech recognition result according to the confusion sample mark.

7. The voice data processing method based on privacy protection according to claim 1 is characterized in that: After the pre-trained acoustic federated learning model performs recognition processing on the combined speech data to obtain a speech recognition result, the method further includes: Converting the speech recognition result into encrypted text; The encrypted text is sent to a preset network address.

8. A voice data processing device based on privacy protection, characterized in that: include: A first processing module, used for acquiring a speech signal to be processed; A second processing module, used for preprocessing the speech signal to be processed to obtain first speech data; A third processing module, configured to determine private data from the first voice data; and to create corresponding obfuscated sample data according to the private data; A fourth processing module, used for combining the obfuscated sample data with the first voice data to obtain combined voice data; The fifth processing module is used to perform recognition processing on the combined speech data based on a pre-trained acoustic federated learning model to obtain a speech recognition result.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the privacy protection-based voice data processing method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that: The computer executable instructions are used to execute the voice data processing method based on privacy protection as described in any one of claims 1 to 7.

Images