Voice data privacy protection method and device, electronic equipment and storage medium

Abstract

The invention provides a voice data privacy protection method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining to-be-identified voice data; extracting local features of the voice data based on a first model, and extracting global features of the voice data based on a second model; fusing the local features and the global features to obtain fused features; and based on a trained voice authentic identification model and the fusion feature, performing synthetic voice authentic identification on the voice data, wherein the voice authentic identification model, the first model and the second model are obtained through synchronous training based on sample voice data, and the first model and the second model are obtained through fine adjustment of a first pre-training model and a second pre-training model respectively. According to the voice data privacy protection method and device, the electronic equipment and the storage medium provided by the invention, the recognition accuracy and robustness of the model on unknown synthetic voice are improved, and unknown voice synthesis attacks are effectively prevented.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a voice data privacy protection method, device, electronic device and storage medium. Background Art

[0002] Speech synthesis technology uses machines to synthesize realistic speech that is similar to that of a speaker. It is widely used in various voice application scenarios such as intelligent voice assistants and disability assistance systems, providing convenience for users. However, the development of speech synthesis technology has also led to security issues such as speech synthesis attacks. Speech synthesis attacks use artificially synthesized speech to imitate the speaker's voice and impersonate the speaker, thereby bypassing the voiceprint authentication system or committing fraud, posing a threat to user privacy. With the application of deep learning technology in the field of speech synthesis, artificially synthesized speech is becoming increasingly difficult to distinguish, and how to prevent speech synthesis attacks has become an important issue.

[0003] For speech synthesis attacks, existing privacy protection methods include preprocessing the voice data before publishing it and detecting the authenticity of synthesized voices. However, for the latest zero-resource speech synthesis attacks, methods based on data preprocessing are not feasible, and methods based on voice authentication cannot effectively prevent them, resulting in poor privacy protection. Summary of the invention

[0004] The present invention provides a voice data privacy protection method, device, electronic device and storage medium, which are used to solve the defects in the prior art that the method based on data preprocessing is not feasible, and the method based on voice authentication cannot effectively prevent, resulting in poor privacy protection effect.

[0005] The present invention provides a voice data privacy protection method, comprising: Acquiring voice data to be identified; Extracting local features of the speech data based on the first model, and extracting global features of the speech data based on the second model; Fusing the local feature with the global feature to obtain a fused feature; Based on the trained speech authentication model and the fusion features, performing synthetic speech authentication on the speech data; Among them, the speech authentication model and the first model and the second model are obtained by synchronous training based on sample speech data, and the first model and the second model are obtained by fine-tuning the first pre-trained model and the second pre-trained model respectively.

[0006] According to the voice data privacy protection method provided by the present invention, the local feature and the global feature are fused to obtain the fused feature, including: The local features and the global features are spliced ​​together, and the spliced ​​features are used as the fusion features.

[0007] According to the voice data privacy protection method provided by the present invention, the local feature and the global feature are concatenated, including: Based on the splicing weight, the local feature and the global feature are spliced, and the splicing weight represents the proportion of the local feature and the global feature in the fusion feature.

[0008] According to the voice data privacy protection method provided by the present invention, the extracting the local features of the voice data based on the first model includes: Encoding the speech data based on the convolution encoder in the first model to obtain a first feature representation sequence; Based on the quantization module in the first model, the first feature representation sequence is mapped to a discrete vector space to obtain a first discrete representation; Based on the transformation module in the first model, the context information of the first discrete representation is extracted through a self-attention mechanism to obtain the local features containing the context information.

[0009] According to the voice data privacy protection method provided by the present invention, the extracting the global features of the voice data based on the second model includes: Encoding the speech data based on a convolutional encoder in a second model to obtain a second feature representation sequence; Clustering the second feature representation sequence based on the acoustic unit detection module in the second model to obtain a category label feature; Based on the transformation module in the second model, the context information of the category label feature is extracted through the self-attention mechanism to obtain the global feature containing the context information.

[0010] According to the voice data privacy protection method provided by the present invention, the synthetic voice authentication of the voice data based on the trained voice authentication model and the fusion feature includes: Based on the speech authentication model, the fusion features are converted into time domain graph features and frequency domain graph features, and the time domain graph features and the frequency domain graph features are merged to obtain merged graph features; Based on the merge graph features, the voice data is subjected to synthetic sound authentication.

[0011] According to the voice data privacy protection method provided by the present invention, the step of obtaining the sample voice data includes: Obtaining initial sample voice data; Generate synthesized speech data based on the speech text corresponding to the initial sample speech data and the timbre corresponding to the initial sample speech data; The sample voice data is determined based on the synthesized voice data and the initial sample voice data.

[0012] The present invention also provides a voice data privacy protection device, comprising: A data acquisition unit, used for acquiring voice data to be identified; A feature extraction unit, configured to extract local features of the speech data based on the first model, and extract global features of the speech data based on the second model; A feature fusion unit, used for fusing the local feature with the global feature to obtain a fused feature; A synthetic voice authentication unit, configured to perform synthetic voice authentication on the voice data based on a trained voice authentication model and the fusion feature; Among them, the speech authentication model and the first model and the second model are obtained by synchronous training based on sample speech data, and the first model and the second model are obtained by fine-tuning the first pre-trained model and the second pre-trained model respectively.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the voice data privacy protection method described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the voice data privacy protection method as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the voice data privacy protection method as described above is implemented.

[0016] The voice data privacy protection method, device, electronic device and storage medium provided by the present invention extract local features and global features of voice data from multiple dimensions, fuse the local features and global features, and perform synthetic sound authentication on voice data based on the fused features, thereby improving the context learning ability and fine-grained information capture ability of the authentication model, improving the accuracy and robustness of the model in recognizing unknown synthetic sounds, and effectively preventing unknown voice synthesis attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is one of the flow charts of the voice data privacy protection method provided by the present invention.

[0019] Figure 2 This is the second flow chart of the voice data privacy protection method provided by the present invention.

[0020] Figure 3 This is one of the structural schematic diagrams of the voice data privacy protection device provided by the present invention.

[0021] Figure 4 This is the second structural schematic diagram of the voice data privacy protection device provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] In response to speech synthesis attacks, existing privacy protection methods can be divided into: pre-processing of speech data before release and synthetic speech authentication.

[0025] The data is preprocessed, including a voice anonymization framework, to prevent attackers from extracting valid voiceprint privacy information from voice data for unauthorized speech synthesis. Voice data information is divided into voiceprint identity information and semantic content information. Under the voice anonymization framework, the identity information and semantic information in the voice data are separated, and the voiceprint identity information is represented as a voiceprint vector. By retaining the semantic content information in the voice data and replacing the voiceprint vector, the voice data identity information is stripped, thereby achieving voice anonymization and preventing unauthorized speech synthesis and speaker identity impersonation.

[0026] Preprocessing the data also includes adding noise to the data to prevent unauthorized speech synthesis. Traditional speech synthesis technology is based on sufficient speaker speech samples to extract effective voiceprint information from the samples to achieve user voiceprint cloning and speech generation. By adding noise that is not recognizable to the human ear but affects the speech generation quality of the speech synthesis model, it prevents attackers from performing speech synthesis attacks without affecting the availability of the speech data.

[0027] On the other hand, synthetic voice authentication aims to identify the received voice data and determine whether the voice data is synthetic voice data. If it is real data, it will continue to be received. If it is synthetic voice data, it will be intercepted to prevent the success of voice synthesis attacks.

[0028] Generally, the successful implementation of speech synthesis attacks requires the attacker to obtain enough speaker speech samples. Existing preprocessing-based solutions prevent attackers from obtaining enough speech samples by performing noise and other processing on the data. However, with the application of large models in speech synthesis technology, zero-resource speech synthesis technology has been realized. Through zero-resource speech synthesis technology, effective speech synthesis no longer requires a large number of speaker speech samples. Only 10 seconds of speaker speech is needed to successfully clone the speaker's voiceprint and synthesize sufficiently realistic speech for identity impersonation. The emergence of zero-resource speech synthesis technology has greatly reduced the cost of speech synthesis attacks and increased the requirements for privacy protection technology. Since the number of samples required to implement zero-resource speech synthesis attacks is too small, and speech data is often used in public platforms such as social networks, methods based on data preprocessing require effective preprocessing of all speech data released by the speaker, which is not feasible.

[0029] On the other hand, although the method based on voice authentication does not require data preprocessing and only requires the deployment of authentication devices at the data receiving end, the cost is relatively low and it is feasible to prevent zero-resource speech synthesis attacks. However, the existing voice authentication model performs much worse than the traditional synthetic voice authentication in the zero-resource synthetic voice authentication task, and cannot effectively authenticate zero-resource synthetic attacks and provide sufficient privacy protection.

[0030] In short, for the latest zero-resource speech synthesis attack, methods based on data preprocessing are not feasible, and methods based on speech authentication cannot effectively prevent it.

[0031] Based on the above problems, an embodiment of the present invention provides a method for protecting the privacy of speech data. In the method, local features of speech data to be identified are extracted based on a first model, and global features of speech data are extracted based on a second model; the local features and the global features are fused to obtain fused features; based on the trained speech authentication model and the fused features, the speech data is authenticated by synthetic sound, wherein the speech authentication model and the first model and the second model are obtained by synchronous training based on sample speech data, and the first model and the second model are obtained by fine-tuning the first pre-trained model and the second pre-trained model respectively.

[0032] This method extracts local and global features of speech data from multiple dimensions, fuses the local and global features, and performs synthetic sound authentication on speech data based on the fused features, thereby improving the context learning ability and fine-grained information capture ability of the authentication model, and enhancing the accuracy and robustness of the model in recognizing unknown synthetic sounds, thus effectively preventing unknown speech synthesis attacks.

[0033] The embodiments of the present invention can be applied to scenarios where voice data privacy protection is required, such as voice data receiving scenarios. The executor of the method can be an electronic device such as a terminal device, a computer, a server, a server cluster, or a specially designed device, or it can be a voice data privacy protection device set in the electronic device. The device can be implemented by software, hardware, or a combination of both.

[0034] In the description of the embodiments of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0035] Figure 1 This is one of the flow charts of the voice data privacy protection method provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 110, obtaining voice data to be identified; Step 120, extracting local features of the speech data based on the first model, and extracting global features of the speech data based on the second model; Step 130, fusing the local features with the global features to obtain fused features; Step 140, performing synthetic sound authentication on the speech data based on the trained speech authentication model and fusion features; Among them, the speech authentication model and the first model and the second model are obtained by synchronous training based on sample speech data, and the first model and the second model are obtained by fine-tuning the first pre-trained model and the second pre-trained model respectively.

[0036] Specifically, the voice data to be identified is the voice that needs to be subjected to synthetic voice authentication, that is, to identify whether the voice data is real data or synthetic data.

[0037] Here, the first model is used to extract local features of speech data, which contain more fine-grained information. The second model is used to extract global features of speech data, which contain global structural information and speech information. Local features and global features can be self-supervisory features of different dimensions and levels.

[0038] Then, the local features and the global features are fused to obtain the fused features. The fused features thus obtained contain multi-dimensional and multi-level speech features to enhance the fine-grained information learning ability and contextual information learning ability of the speech authentication model, thereby improving the accuracy and robustness of the speech authentication model.

[0039] Here, the first model, the second model and the speech authentication model can be trained in advance, the speech authentication model and the first model and the second model are obtained by synchronous training based on sample speech data, and the first model and the second model are obtained by fine-tuning the first pre-trained model and the second pre-trained model respectively. Further, the first pre-trained model can be wav2vec2.0, after fine-tuning, the first model is obtained, and the second pre-trained model can be a HuBERT model, after fine-tuning, the second model is obtained.

[0040] In some embodiments, extracting local features of the speech data based on the first model in step 120 includes: Step 121, encoding the speech data based on the convolution encoder in the first model to obtain a first feature representation sequence; Step 122, based on the quantization module in the first model, mapping the first feature representation sequence to a discrete vector space to obtain a first discrete representation; Step 123: Based on the transformation module in the first model, the context information of the first discrete representation is extracted through the self-attention mechanism to obtain local features containing the context information.

[0041] Specifically, the first model uses a convolutional encoder to encode the input speech data, capture the local temporal dependencies of the speech signal, and generate a low-level feature representation sequence. X w2v , that is, the first feature representation sequence. The quantization module converts the first feature representation sequence X w2vMapping to discrete vector space, obtaining discrete representation of the sequence Q , which is the first discrete representation. Discrete representation Q is fed into a Transformer-based network that captures long-range dependencies via self-attention to generate high-level, contextual feature representations O w2v After repeated processing by the transformer network, the first model outputs a feature vector that captures local fine-grained information and contains rich contextual information. F w2v ， That is, local features that contain contextual information.

[0042] In some embodiments, extracting global features of the speech data based on the second model in step 120 specifically includes: Step 124, encoding the speech data based on the convolution encoder in the second model to obtain a second feature representation sequence; Step 125, clustering the second feature representation sequence based on the acoustic unit detection module in the second model to obtain a category label feature; Step 126, based on the transformation module in the second model, extract the context information of the category label feature through the self-attention mechanism to obtain the global feature containing the context information.

[0043] Specifically, the second model uses a convolutional encoder to encode the input speech data, capture the local temporal dependencies of the speech signal, and generate a low-level feature representation sequence. X HuB , i.e., the second feature representation sequence. The acoustic unit detection module detects the second feature representation sequence X HuB Implement k-means clustering operation to convert continuous features into discrete category label features Z , these labels represent the potential structural information of the audio data. Category label features Z is fed into a Transformer-based network that captures long-range dependencies via self-attention to generate high-level, contextual feature representations O HuB 。 After repeated processing by the transformer network, the second model outputs a feature vector that captures global structural and semantic information and contains rich contextual information. F HuB , which is the global feature.

[0044] In some embodiments, in step 130, the local features and the global features are fused to obtain the fused features, which specifically includes: splicing the local features and the global features, and using the spliced ​​features as the fused features.

[0045] Specifically, the extracted global features and local features are concatenated, that is, they are combined into a new feature vector in a certain order (such as global first, then local or by feature dimension). When concatenating, attention should be paid to the dimensional consistency of the feature vectors. If the dimensions are different, appropriate transformation or padding may be required.

[0046] For example, assuming that the dimension of the global feature vector is N and the dimension of the local feature vector is M, the dimension of the fused feature vector after concatenation is N + M. The concatenation operation can be implemented by simple matrix merging.

[0047] In some other embodiments, feature stitching the local features and the global features specifically includes: based on a stitching weight, feature stitching the local features and the global features, the stitching weight represents the proportion of the local features and the global features in the fused features.

[0048] Fusion Features F , the specific calculation process is as follows:

[0049] in, F w2v is the self-supervised speech feature extracted by the first model, i.e., the local feature; F HuB is the self-supervised speech feature extracted by the second model, that is, the global feature. W is the splicing weight, which represents the proportion of local features and global features in the fusion features. The splicing weight here is a learnable weight, which is iteratively optimized and updated synchronously during model training and fine-tuning to balance the local features and global features in the output fusion features. F The weight ratio in . Fusion features F It has local, fine-grained information extracted by the first model and global, hierarchical information extracted by the second model.

[0050] In some embodiments, based on the trained speech authentication model and fusion features, the speech data is authenticated by synthetic speech, that is, step 140 specifically includes: Step 141, based on the speech authentication model, converting the fusion features into time domain features and frequency domain features, and merging the time domain features and the frequency domain features to obtain a merged feature; Step 142: Based on the merged graph features, the voice data is subjected to synthetic sound authentication.

[0051] Specifically, the neural network based on the AASIST authentication model is used to learn the input fusion features in a more fine-grained manner and complete the synthetic sound authentication task of speech data. The fusion feature vector F is input, and the AASIST authentication model converts it into time domain graph features.G t Frequency domain features G f , and merge the time domain graph features and the frequency domain graph features to obtain the merged graph features G ft . For the merged graph features G ft Graph neural networks and graph attention mechanisms are used for feature learning and binary classification authentication. The classification labels are real data and synthetic data, and the identification of synthetic sounds is completed based on the classification results.

[0052] Based on any of the above embodiments, the step of acquiring the sample voice data includes: Obtaining initial sample voice data; Generate synthesized speech data based on the speech text corresponding to the initial sample speech data and the timbre corresponding to the initial sample speech data; Based on the synthesized speech data and the initial sample speech data, sample speech data is determined.

[0053] Specifically, the initial sample voice data may be data in the ASVspoof2019 LA voice authentication dataset. In order to improve the accuracy of the model in recognizing zero-resource synthesized voices and effectively prevent unknown zero-resource voice synthesis attacks, this embodiment uses zero-resource voice synthesis technology to expand the initial sample voice data.

[0054] Perform voice cloning for each speaker in the LA dataset. Randomly select the speech text in the LA dataset, that is, the speech text corresponding to the initial sample speech data, perform speech synthesis based on the cloned speaker's timbre, generate zero-resource synthetic speech data equal to the speech data of each speaker in the original LA dataset, and obtain the expanded LAplus dataset. The expanded LAplus dataset includes synthetic speech data and initial sample speech data.

[0055] Based on any of the above embodiments, Figure 2 This is the second flow chart of the voice data privacy protection method provided by the present invention. Figure 2 As shown, a voice data privacy protection method is provided, comprising: The first step S11 is to expand the data set. Before training the speech authentication model, a zero-resource speech synthesis model is used to generate zero-resource synthetic sound data, and the speech authentication data set is expanded so that the model can effectively learn the data features of zero-resource synthetic sound. Specifically, it includes: obtaining the initial sample speech data; generating synthetic speech data based on the speech text corresponding to the initial sample speech data and the timbre corresponding to the initial sample speech data; and determining the sample speech data based on the synthetic speech data and the initial sample speech data.

[0056] The second step S12 is data set preprocessing. Before training the fake detection model, it is necessary to perform data enhancement on the training data, adding linear convolution noise, nonlinear convolution noise and pulse signal additive noise to each data to generate training data that can effectively improve the fake detection model's ability to detect unknown synthetic sounds.

[0057] The third step S13 is fusion feature extraction. Two pre-trained models are used to extract self-supervisory features of different dimensions from the input speech data, and the two features are effectively spliced ​​by adding learnable weights to obtain multi-dimensional and multi-level speech features to improve the fine-grained information learning ability and contextual information learning ability of the false identification model, thereby improving the accuracy and robustness of the false identification model. The learnable weights need to be combined with the false identification model training in the fourth and fifth steps and the pre-trained model fine-tuning, and the parameters are updated synchronously.

[0058] Construct self-supervised features based on the first pre-trained model wav2vec 2.0 and the second pre-trained model HuBERT, which can effectively improve the accuracy and robustness of the anti-counterfeiting model. The pre-processed speech data will be input into the wav2vec2.0 and HuBERT pre-trained models for feature learning, and then the extracted features will be weighted and concatenated based on the learnable weights to output a multi-dimensional and more fine-grained speech data feature representation, namely the fusion feature.

[0059] Step 4 S14: Training of the fake detection model. The concatenated fusion features will be input into the neural network model to complete further feature extraction. Through model training, the fake detection model learns the ability to distinguish between synthetic sound data and real data. During training, the same model learning rate lr as the first pre-training model and the second pre-training model is set, and the enhanced LAplus training set is used for training. The weight cross entropy is used to calculate the loss value, and the back propagation algorithm is used to update the weight matrix in the AASIST model.

[0060] Step 5 S15: Fine-tune the pre-trained model. Fine-tune the pre-trained model used in step 3, and update the parameters synchronously in combination with the false positive detection model training in step 4. When fine-tuning the wav2vec 2.0 pre-trained model and the HuBERT pre-trained model, they need to be trained synchronously in combination with the AASIST false positive detection model used. Use weighted cross entropy to calculate the loss value, and use the back propagation algorithm to update the wav2vec 2.0 model parameters and the HuBERT model parameters. Repeat steps 3 to 5 until the model converges and the training is completed.

[0061] Step 6 S16 Synthetic voice identification. First, the voice data to be identified is preprocessed, mainly including resampling and length truncation. Then, the preprocessed voice data is identified with the voice authentication model trained in the third and fourth steps, and the identification result is output.

[0062] The method provided by the embodiment of the present invention improves the accuracy of the model in identifying zero-resource synthetic sounds by expanding the zero-resource synthetic sound data to train the authentication model, and effectively prevents unknown zero-resource speech synthesis attacks. By effectively splicing multiple self-supervisory features, the detailed features and global features of speech data are extracted from multiple dimensions, the context learning ability and fine-grained information capture ability of the authentication model are improved, the accuracy and robustness of the model in identifying unknown synthetic sounds are improved, and the unknown speech synthesis attacks are effectively prevented.

[0063] Based on any of the above embodiments, Figure 3 FIG. 1 is one of the structural diagrams of the voice data privacy protection device provided by the present invention. Figure 3 As shown, a voice data privacy protection device is provided, comprising: The data acquisition unit 310 is used to acquire the voice data to be identified; A feature extraction unit 320, configured to extract local features of the speech data based on the first model, and extract global features of the speech data based on the second model; A feature fusion unit 330, used to fuse the local feature with the global feature to obtain a fused feature; A synthetic voice identification unit 340, configured to perform synthetic voice identification on the voice data based on a trained voice identification model and the fusion feature; Among them, the speech authentication model and the first model and the second model are obtained by synchronous training based on sample speech data, and the first model and the second model are obtained by fine-tuning the first pre-trained model and the second pre-trained model respectively.

[0064] Based on any of the above embodiments, the feature fusion unit is specifically used for: The local features and the global features are spliced ​​together, and the spliced ​​features are used as the fusion features.

[0065] Based on any of the foregoing embodiments, the feature fusion unit is specifically used for: Based on the splicing weight, the local feature and the global feature are spliced, and the splicing weight represents the proportion of the local feature and the global feature in the fusion feature.

[0066] Based on any of the above embodiments, the feature extraction unit is specifically used for: Encoding the speech data based on the convolution encoder in the first model to obtain a first feature representation sequence; Based on the quantization module in the first model, the first feature representation sequence is mapped to a discrete vector space to obtain a first discrete representation; Based on the transformation module in the first model, the context information of the first discrete representation is extracted through a self-attention mechanism to obtain the local features containing the context information.

[0067] Based on any of the above embodiments, the feature extraction unit is specifically used for: Encoding the speech data based on a convolutional encoder in a second model to obtain a second feature representation sequence; Clustering the second feature representation sequence based on the acoustic unit detection module in the second model to obtain a category label feature; Based on the transformation module in the second model, the context information of the category label feature is extracted through the self-attention mechanism to obtain the global feature containing the context information.

[0068] Based on any of the above embodiments, the synthetic sound authentication unit is specifically used for: Based on the speech authentication model, the fusion features are converted into time domain graph features and frequency domain graph features, and the time domain graph features and the frequency domain graph features are merged to obtain merged graph features; Based on the merge graph features, the voice data is subjected to synthetic sound authentication.

[0069] Based on any of the above embodiments, it further includes a sample data acquisition unit, which is used to: Obtaining initial sample voice data; Generate synthesized speech data based on the speech text corresponding to the initial sample speech data and the timbre corresponding to the initial sample speech data; The sample voice data is determined based on the synthesized voice data and the initial sample voice data.

[0070] Based on any of the above embodiments, Figure 4 This is the second structural diagram of the voice data privacy protection device provided by the present invention. Figure 4 As shown, a voice data privacy protection device is provided, including: LAplus data set expansion module, LAplus data set preprocessing module, wav2vec2.0 voice feature extraction module, HuBERT voice feature extraction module, voice feature splicing module, neural network voice authentication module, pre-trained model fine-tuning and authentication model training module, voice data preprocessing module, and synthetic voice authentication module. The workflow of each module is as follows: In the LAplus dataset expansion module, the ASVspoof2019 LA training dataset is input, the LA training set data is expanded with zero-resource synthetic sound, the expanded LAplus dataset is generated, and it is output to the LAplus dataset preprocessing module.

[0071] In the LAplus dataset preprocessing module, the input LAplus dataset will be enhanced by adding linear convolution noise, nonlinear convolution noise and pulse signal additive noise to it, and then output to the wav2vec 2.0 and HuBERT speech feature extraction modules.

[0072] In the wav2vec2.0 speech feature extraction module, the preprocessed LAplus dataset is input, the speech data features of the dataset are extracted using the wav2vec2.0 xslr (0.3B) pre-trained model, and then output to the speech feature splicing module.

[0073] In the HuBERT speech feature extraction module, the preprocessed LAplus dataset is input, the speech data features of the dataset are extracted using the HuBERT base pre-trained model, and then output to the speech feature splicing module.

[0074] In the speech feature splicing module, the speech data features extracted by the wav2vec2.0 and HuBERT models are input, the two features are spliced ​​using learnable weights, and the spliced ​​multi-dimensional features are output to the neural network speech authentication module.

[0075] In the neural network speech authentication module, the splicing features are input for further feature extraction and classification. The neural network classification model in the module is based on the AASIST authentication model structure, and uses the graph neural network structure and graph attention mechanism to learn more fine-grained speech data features, and finally completes the synthetic sound identification, calculates the loss value and outputs it to the pre-trained model fine-tuning and model training module.

[0076] In the pre-trained model fine-tuning and speech detection model training modules, appropriate learning rates are set for the pre-trained model and speech detection model respectively, and parameters are updated according to the input loss value. Training is iterated until convergence, and finally the trained synthetic speech detection model is output.

[0077] In the speech data preprocessing module, the target speech data to be identified is input, data truncation and resampling operations are performed on it, and it is output to the synthetic sound identification module.

[0078] In the synthetic sound identification module, the preprocessed target speech data and the synthetic sound identification model output from the pre-trained model fine-tuning and identification model training module are input to complete the synthetic sound identification of the target speech data and output the identification result.

[0079] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the voice data privacy protection method, which includes: obtaining voice data to be identified; extracting local features of the voice data based on a first model, and extracting global features of the voice data based on a second model; fusing the local features with the global features to obtain fused features; performing synthetic voice authentication on the voice data based on a trained voice authentication model and the fused features; wherein the voice authentication model and the first model and the second model are obtained by synchronous training based on sample voice data, and the first model and the second model are obtained by fine-tuning the first pre-trained model and the second pre-trained model respectively.

[0080] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0081] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the voice data privacy protection method provided by the above-mentioned methods, which includes: obtaining voice data to be identified; extracting local features of the voice data based on a first model, and extracting global features of the voice data based on a second model; fusing the local features with the global features to obtain fused features; performing synthetic sound authentication on the voice data based on a trained voice authentication model and the fused features; wherein the voice authentication model and the first model and the second model are obtained by synchronous training based on sample voice data, and the first model and the second model are obtained by fine-tuning the first pre-trained model and the second pre-trained model respectively.

[0082] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the voice data privacy protection method provided by the above-mentioned methods, the method comprising: obtaining voice data to be identified; extracting local features of the voice data based on a first model, and extracting global features of the voice data based on a second model; fusing the local features with the global features to obtain fused features; performing synthetic sound authentication on the voice data based on a trained voice authentication model and the fused features; wherein the voice authentication model and the first model and the second model are obtained by synchronous training based on sample voice data, and the first model and the second model are obtained by fine-tuning the first pre-trained model and the second pre-trained model respectively.

[0083] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0084] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A voice data privacy protection method, characterized in that: include: Acquiring voice data to be identified; Extracting local features of the speech data based on the first model, and extracting global features of the speech data based on the second model; Fusing the local feature with the global feature to obtain a fused feature; Based on the trained speech authentication model and the fusion features, performing synthetic speech authentication on the speech data; Among them, the speech authentication model and the first model and the second model are obtained by synchronous training based on sample speech data, and the first model and the second model are obtained by fine-tuning the first pre-trained model and the second pre-trained model respectively.

2. The voice data privacy protection method according to claim 1, characterized in that: The fusing the local features and the global features to obtain fused features includes: The local features and the global features are spliced ​​together, and the spliced ​​features are used as the fusion features.

3. The voice data privacy protection method according to claim 2, characterized in that: The step of performing feature splicing on the local feature and the global feature comprises: Based on the splicing weight, the local feature and the global feature are spliced, and the splicing weight represents the proportion of the local feature and the global feature in the fusion feature.

4. The voice data privacy protection method according to claim 1, characterized in that: The extracting the local features of the speech data based on the first model comprises: Encoding the speech data based on the convolution encoder in the first model to obtain a first feature representation sequence; Based on the quantization module in the first model, the first feature representation sequence is mapped to a discrete vector space to obtain a first discrete representation; Based on the transformation module in the first model, the context information of the first discrete representation is extracted through a self-attention mechanism to obtain the local features containing the context information.

5. The voice data privacy protection method according to claim 1, characterized in that: The extracting the global features of the speech data based on the second model comprises: Encoding the speech data based on the convolutional encoder in the second model to obtain a second feature representation sequence; Clustering the second feature representation sequence based on the acoustic unit detection module in the second model to obtain a category label feature; Based on the transformation module in the second model, the context information of the category label feature is extracted through the self-attention mechanism to obtain the global feature containing the context information.

6. The voice data privacy protection method according to any one of claims 1 to 5, characterized in that: The performing synthetic sound authentication on the speech data based on the trained speech authentication model and the fusion feature includes: Based on the speech authentication model, the fusion features are converted into time domain features and frequency domain features, and the time domain features and the frequency domain features are merged to obtain merged features; Based on the merge graph features, the voice data is subjected to synthetic sound authentication.

7. The voice data privacy protection method according to any one of claims 1 to 5, characterized in that: The step of acquiring the sample voice data comprises: Obtaining initial sample voice data; Generate synthesized speech data based on the speech text corresponding to the initial sample speech data and the timbre corresponding to the initial sample speech data; The sample voice data is determined based on the synthesized voice data and the initial sample voice data.

8. A voice data privacy protection device, characterized in that: include: A data acquisition unit, used for acquiring voice data to be identified; A feature extraction unit, configured to extract local features of the speech data based on the first model, and extract global features of the speech data based on the second model; A feature fusion unit, used for fusing the local feature with the global feature to obtain a fused feature; A synthetic voice authentication unit, configured to perform synthetic voice authentication on the voice data based on a trained voice authentication model and the fusion feature; Among them, the speech authentication model and the first model and the second model are obtained by synchronous training based on sample speech data, and the first model and the second model are obtained by fine-tuning the first pre-trained model and the second pre-trained model respectively.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the voice data privacy protection method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the voice data privacy protection method according to any one of claims 1 to 7 is implemented.

Images