Sound information protection method and device, storage medium, and electronic device

Abstract

Embodiments of the present application provide a sound information protection system and method, a storage medium and an electronic device. The method comprises: after determining that a current voice incoming call belongs to the target call and obtaining an instruction to perform sound conversion, converting the sound of the current user into the sound of the target speaker through a lightweight voice conversion model, and then calling the current incoming call party. The lightweight voice conversion model is a model trained using sample voice containing the voice of the target speaker, and comprises a posterior encoder, a prior encoder and a decoder. The problem of how to avoid the leakage of user voice features in the related art is solved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a method and apparatus for protecting sound information, a storage medium, and an electronic device. Background Technology

[0002] Currently, the main approach to voice conversion in related technologies involves first using Automatic Speech Recognition (ASR) to identify the speech to be converted and extract Backus-Naur form (BNF) features. Then, Text-to-Speech (TTS) technology is used to synthesize the target speech based on the identified content and BNF features. This process is limited by the accuracy and efficiency of ASR, resulting in unsatisfactory accuracy and real-time performance in practical applications. To address this, end-to-end voice conversion methods have been proposed, which directly convert the voice of the speech to be converted without using ASR. However, end-to-end voice conversion typically relies on large network models, leading to poor real-time performance in actual conversion processes.

[0003] Currently, the aforementioned voice conversion technology is frequently used by criminals for telecommunications fraud. When committing telecommunications fraud against a victim, criminals obtain voice samples from individuals connected to the victim through various channels, such as recordings of phone conversations between the victim's relatives and third parties. After obtaining these voice samples, the criminals use voice conversion technology to communicate with the victim using the voice of the person in question, thereby achieving their illegal purposes. Although policies and regulations have made certain provisions regarding the leakage of personal information, personal voice characteristics are more abstract than traditional personal information, and the leakage is more covert, making it difficult to completely prevent the leakage of voice characteristics during voice communications between individuals and third parties.

[0004] Among related technologies, those for preventing telecom fraud based on voice conversion mainly focus on detecting whether voice conversion technology is used during the telecom fraud process, but no effective solution has yet been proposed for the problem of how to avoid the leakage of users' voice characteristics. Summary of the Invention

[0005] This application provides a method and apparatus for protecting voice information, a storage medium, and an electronic device to at least solve the problem of how to avoid the leakage of user voice characteristics in related technologies.

[0006] In one embodiment of this application, a method for protecting voice information is provided, comprising: determining whether a current incoming voice call belongs to a target call requiring voice conversion; wherein the target call includes at least: tagged harassing calls, sales calls, and customer service calls, as well as calls from which the caller is not recorded in the current terminal's address book; after determining that the current incoming voice call belongs to the target call and obtaining an instruction to perform voice conversion, converting the current user's voice into the voice of a target speaker using a lightweight voice conversion model and then conducting a call with the current caller; wherein the target speaker is a random third party or is specified by the current user, and the lightweight voice conversion model is a model trained using sample speech containing the voice of the target speaker, comprising: a posterior encoder, a prior encoder, and a decoder.

[0007] In one embodiment, before engaging in a call with the current caller after converting the current user's voice into the target speaker's voice using a lightweight speech conversion model, the method further includes: identifying the style category of the target speaker's voice sample using an audio feature encoding module, and extracting audio features of the target speaker's voice sample based on the style category; wherein the audio features of the target speaker's voice sample include text features, prosodic features, and timbre features; and extracting audio features corresponding to the current user's voice using the audio feature encoding module, wherein the audio features corresponding to the current user's voice include text features, prosodic features, and timbre features of the current user's voice.

[0008] In one embodiment, the step of converting the current user's voice into the target speaker's voice using a lightweight speech conversion model and then communicating with the current caller includes: fusing the audio features of the current user's speech with the audio features of the target speaker's speech sample using the posterior encoder to obtain joint encoded features, and extracting a first implicit feature based on the audio features of the target speaker's speech sample; obtaining the prior distribution of the first implicit feature from the phonemes of the current user's speech using the prior encoder based on the posterior probability map (PPG) features of the target speaker's speech sample through a linear attention mechanism; wherein the prior distribution is characterized by the mean or variance corresponding to the first implicit feature obtained with the PPG features as constraints; and decoding the joint encoded features by the decoder based on the prior distribution output by the prior encoder using a standard streaming operation, and outputting the converted speech through a vocoder; wherein the decoder includes a high-frequency decoding module and a low-frequency decoding module, the high-frequency decoding module using an inverse short-time Fourier transform for decoding, and the low-frequency decoding module using a vocoder for decoding.

[0009] In one embodiment, the step of obtaining the prior distribution of the first implicit feature from the phonemes of the current user's speech through the prior encoder based on the posterior probability map (PPG) features of the target speaker's speech sample using a linear attention mechanism includes: pre-training a speech recognition model independent of the prior encoder; wherein the training sample data of the speech recognition model does not include the speech sample of the target speaker; extracting the PPG features of the target speaker's speech sample through the trained speech recognition model; and calculating the constraint loss in the constraint process through a PPG feature prediction unit set independently of the prior encoder, and compensating for the extracted PPG features.

[0010] In one embodiment, after obtaining the prior distribution of the first implicit feature from the phonemes of the current user's speech through the prior encoder based on the posterior probability map (PPG) features of the target speaker's speech sample using a linear attention mechanism, the method further includes: complicating the prior distribution of the first implicit feature through a normalization layer in the prior encoder and outputting the corresponding spectral information; wherein the normalization layer is composed of multiple coupling layers, each of the coupling layers is composed of multiple residual blocks, and the multiple coupling layers share parameters.

[0011] In one embodiment, the lightweight speech conversion model includes the following steps during training: acquiring the audio features of the target speaker's speech sample and its corresponding second implicit features; generating a predicted value of the target speaker's speech sample based on the audio features and the second implicit features; training a first loss function based on the predicted value and the true value of the target speaker's speech sample; wherein the first loss function indicates the loss of the predicted value of the target speaker's speech sample relative to the true value; aligning the audio features and the second implicit features of the target speaker's speech sample using an alignment prediction unit to calculate an alignment matrix, and training a second loss function based on the alignment matrix, wherein the second loss function indicates the dispersion between the audio features and the second implicit features of the target speaker's speech sample; determining a discrimination result based on the difference between the predicted value and the true value of the target speech sample, and training a third loss function based on the discrimination result; and obtaining the trained lightweight speech conversion model based on the first loss function, the second loss function, and the third loss function.

[0012] In one embodiment, before extracting the audio features corresponding to the current user's speech through the audio feature encoding module, the method further includes: training a first clustering model using a first training sample, wherein the first training sample includes speech samples from multiple speakers, and the speech samples from multiple speakers correspond to different style types; the first clustering model is used to cluster the first training sample and determine the category label corresponding to the first training sample based on the clustering result; training a second clustering model using a second training sample, wherein the second training sample includes speech samples from multiple speakers, and the speech samples from multiple speakers correspond to different style types; the second clustering model is used to cluster the second training sample and determine the category label corresponding to the second training sample based on the clustering result. The first clustering model and the second clustering model employ feature extractors with different structures to cluster from different dimensions. A third training sample is simultaneously input into the trained first clustering model, the trained second clustering model, and the initial audio feature encoding module, wherein the third training sample contains speech samples from multiple speakers. The initial audio feature encoding module is trained according to its loss function, and simultaneously trained until convergence based on the true class labels output by the first clustering model and the second clustering model, and the predicted class labels output by the initial audio feature encoding module, thus obtaining the audio feature encoding module. The audio feature encoding module is used to extract audio features based on the style type of the speech.

[0013] According to another embodiment of this application, a voice information protection device is also provided, comprising: a determination module configured to determine whether the current incoming voice call belongs to a target call requiring voice conversion; wherein the target call includes at least: tagged harassing calls, sales calls, and customer service calls, as well as calls from which the caller is not recorded in the current terminal's address book; and a conversion module configured to, after determining that the current incoming voice call belongs to the target call and obtaining an instruction to perform voice conversion, convert the current user's voice into the voice of a target speaker using a lightweight voice conversion model and then conduct a call with the current caller; wherein the target speaker is a random third party or is specified by the current user, and the lightweight voice conversion model is a model trained using sample speech containing the voice of the target speaker, including: a posterior encoder, a prior encoder, and a decoder.

[0014] In one embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to execute the steps of any of the above method embodiments at runtime.

[0015] In one embodiment of this application, an electronic device is also provided, including a memory and a processor, characterized in that the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0016] The voice information protection method provided in this application, after determining that the current incoming voice call belongs to the target call and obtaining the instruction to perform voice conversion, converts the current user's voice into the target speaker's voice using a lightweight voice conversion model before communicating with the caller. The lightweight voice conversion model is a model trained using sample voices containing the target speaker's voice, and includes a posterior encoder, a prior encoder, and a decoder. This solves the problem of preventing the leakage of user voice features in related technologies. By identifying the style category of the target voice sample through an audio feature encoding module, it avoids detecting whether voice conversion technology is used during telecommunications fraud, and instead prevents the leakage of user voices at the source, thus preventing fraudsters from obtaining the user's voice information. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of a mobile terminal for a sound information protection method according to an embodiment of this application.

[0019] Figure 2 This is a flowchart of an optional sound information protection method according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the structure of an optional audio feature encoding module according to an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the training process of an optional audio feature encoding module according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of an optional electronic device structure according to an embodiment of this application. Detailed Implementation

[0023] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0025] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for voice information protection according to an embodiment of this application. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0026] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer programs corresponding to the training methods of the speech conversion model, clustering model, audio feature encoding module, style encoding module, or timbre conversion module in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thus implementing the methods described above. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0027] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0028] Figure 2 This is a flowchart of an optional sound information protection method according to an embodiment of this application. In one embodiment of this application, a sound information protection method is provided, including:

[0029] Step S202: Determine whether the current voice call belongs to the target call that needs to be converted; wherein, the target call includes at least: tagged harassing calls, sales calls and customer service calls, as well as calls from the caller that are not recorded in the current terminal's address book;

[0030] Step S204: After determining that the current voice call belongs to the target call and obtaining the instruction to perform voice conversion, the user's voice is converted into the target speaker's voice using a lightweight voice conversion model and then the user communicates with the current caller. The target speaker is a random third party or is specified by the current user. The lightweight voice conversion model is a model trained using sample voices containing the target speaker's voice, including a posterior encoder, a prior encoder, and a decoder.

[0031] It should be noted that phone numbers not recorded in the current device's address book can also be configured accordingly. For example, some users do not add the names or titles of their family members in their address book to protect their privacy. The system can set up a whitelist of numbers that can be called more than 3 or 5 times without requiring confirmation. Alternatively, users can set up their own whitelist of numbers without requiring confirmation.

[0032] In one embodiment, before engaging in a call with the current caller after converting the current user's voice into the target speaker's voice using a lightweight speech conversion model, the method further includes: identifying the style category of the target speaker's voice sample using an audio feature encoding module, and extracting audio features of the target speaker's voice sample based on the style category; wherein the audio features of the target speaker's voice sample include text features, prosodic features, and timbre features; and extracting audio features corresponding to the current user's voice using the audio feature encoding module, wherein the audio features corresponding to the current user's voice include text features, prosodic features, and timbre features of the current user's voice.

[0033] In one embodiment, the step of converting the current user's voice into the target speaker's voice using a lightweight speech conversion model and then communicating with the current caller includes: fusing the audio features of the current user's speech with the audio features of the target speaker's speech sample using the posterior encoder to obtain joint encoded features, and extracting a first implicit feature based on the audio features of the target speaker's speech sample; obtaining the prior distribution of the first implicit feature from the phonemes of the current user's speech using the prior encoder based on the posterior probability map (PPG) features of the target speaker's speech sample through a linear attention mechanism; wherein the prior distribution is characterized by the mean or variance corresponding to the first implicit feature obtained with the PPG features as constraints; and decoding the joint encoded features by the decoder based on the prior distribution output by the prior encoder using a standard streaming operation, and outputting the converted speech through a vocoder; wherein the decoder includes a high-frequency decoding module and a low-frequency decoding module, the high-frequency decoding module using an inverse short-time Fourier transform for decoding, and the low-frequency decoding module using a vocoder for decoding.

[0034] In one embodiment, the step of obtaining the prior distribution of the first implicit feature from the phonemes of the current user's speech through the prior encoder based on the posterior probability map (PPG) features of the target speaker's speech sample using a linear attention mechanism includes: pre-training a speech recognition model independent of the prior encoder; wherein the training sample data of the speech recognition model does not include the speech sample of the target speaker; extracting the PPG features of the target speaker's speech sample through the trained speech recognition model; and calculating the constraint loss in the constraint process through a PPG feature prediction unit set independently of the prior encoder, and compensating for the extracted PPG features.

[0035] In one embodiment, after obtaining the prior distribution of the first implicit feature from the phonemes of the current user's speech through the prior encoder based on the posterior probability map (PPG) features of the target speaker's speech sample using a linear attention mechanism, the method further includes: complicating the prior distribution of the first implicit feature through a normalization layer in the prior encoder and outputting the corresponding spectral information; wherein the normalization layer is composed of multiple coupling layers, each of the coupling layers is composed of multiple residual blocks, and the multiple coupling layers share parameters.

[0036] In one embodiment, the lightweight speech conversion model includes the following steps during training: acquiring the audio features of the target speaker's speech sample and its corresponding second implicit features; generating a predicted value of the target speaker's speech sample based on the audio features and the second implicit features; training a first loss function based on the predicted value and the true value of the target speaker's speech sample; wherein the first loss function indicates the loss of the predicted value of the target speaker's speech sample relative to the true value; aligning the audio features and the second implicit features of the target speaker's speech sample using an alignment prediction unit to calculate an alignment matrix, and training a second loss function based on the alignment matrix, wherein the second loss function indicates the dispersion between the audio features and the second implicit features of the target speaker's speech sample; determining a discrimination result based on the difference between the predicted value and the true value of the target speech sample, and training a third loss function based on the discrimination result; and obtaining the trained lightweight speech conversion model based on the first loss function, the second loss function, and the third loss function.

[0037] In one embodiment, before extracting the audio features corresponding to the current user's speech through the audio feature encoding module, the method further includes: training a first clustering model using a first training sample, wherein the first training sample includes speech samples from multiple speakers, and the speech samples from multiple speakers correspond to different style types; the first clustering model is used to cluster the first training sample and determine the category label corresponding to the first training sample based on the clustering result; training a second clustering model using a second training sample, wherein the second training sample includes speech samples from multiple speakers, and the speech samples from multiple speakers correspond to different style types; the second clustering model is used to cluster the second training sample and determine the category label corresponding to the second training sample based on the clustering result. The first clustering model and the second clustering model employ feature extractors with different structures to cluster from different dimensions. A third training sample is simultaneously input into the trained first clustering model, the trained second clustering model, and the initial audio feature encoding module, wherein the third training sample contains speech samples from multiple speakers. The initial audio feature encoding module is trained according to its loss function, and simultaneously trained until convergence based on the true class labels output by the first clustering model and the second clustering model, and the predicted class labels output by the initial audio feature encoding module, thus obtaining the audio feature encoding module. The audio feature encoding module is used to extract audio features based on the style type of the speech.

[0038] According to another embodiment of this application, a voice information protection device is also provided, comprising: a determination module configured to determine whether the current incoming voice call belongs to a target call requiring voice conversion; wherein the target call includes at least: tagged harassing calls, sales calls, and customer service calls, as well as calls from which the caller is not recorded in the current terminal's address book; and a conversion module configured to, after determining that the current incoming voice call belongs to the target call and obtaining an instruction to perform voice conversion, convert the current user's voice into the voice of a target speaker using a lightweight voice conversion model and then conduct a call with the current caller; wherein the target speaker is a random third party or is specified by the current user, and the lightweight voice conversion model is a model trained using sample speech containing the voice of the target speaker, including: a posterior encoder, a prior encoder, and a decoder.

[0039] It should be noted that the lightweight speech conversion model in this application embodiment mainly consists of three parts: a posterior encoder, a prior encoder, and a decoder.

[0040] 1) Posterior encoder: used to extract the corresponding implicit features from the audio features during the training process.

[0041] 2) Prior encoder: Used to obtain the prior distribution of implicit features from phonemes based on the Phonetic Posteriorgram (PPG) features.

[0042] Specifically, a speech recognition model can be pre-trained. This model is not trained using samples from the target speaker in this embodiment, thus obtaining a speech recognition model independent of the target speaker. After training, the speech recognition unit can extract corresponding PPG features based on the phonemes of the audio features. It should be noted that this speech recognition unit is independent of the prior encoder settings.

[0043] After extracting PPG features using the aforementioned speech recognition model, the prior encoder obtains a prior distribution based on the PPG features and implicit features. Specifically, the prior distribution is represented by the mean or variance of the implicit features obtained with the PPG features as constraints. During the computation process, the prior encoder can also introduce an attention mechanism and employ a linear attention mechanism to reduce computational complexity.

[0044] Considering that implicit features often lack explicit constraints, making them prone to errors in extracting PPG features or confirming prior distributions, this can lead to pronunciation errors and tonal abnormalities in the overall voice conversion. Therefore, a PPG feature prediction unit is added to calculate the constraint loss during the constraint process. The input to the PPG feature prediction unit is the implicit features extracted from other training samples, and the output is the predicted PPG features based on those training samples. By comparing the predicted PPG features with the true PPG features, a constraint loss function can be established. This function can then compensate for the PPG features extracted from the implicit features during the prior encoder's operation. It should be noted that this PPG feature prediction unit is set independently of the prior encoder.

[0045] Subsequently, the prior encoder further complicates the prior distribution of implicit features through a set normalization layer and outputs the corresponding spectral information.

[0046] Specifically, the normalization layer consists of multiple coupling layers, and each coupling layer consists of multiple residual blocks. These coupling layers share parameters, thus allowing for the control of the number of parameters in the flow to a single layer while further complicating the prior distribution of latent features, thereby further controlling the overall number of parameters in the model.

[0047] 3) Decoder: The decoder consists of a high-frequency decoding section and a low-frequency decoding section. The high-frequency decoding section employs an inverse short-time Fourier transform (ISFT) to perform the decoding. This high-frequency decoding section comprises multiple convolutional layers. If the spectral information input from the prior encoder to the decoder corresponds to the dimension N, then the dimension of the multiple convolutional layers in the high-frequency decoding section is 2N*(M / 2+1), where M indicates the absolute value of the IFT, i.e., its amplitude-frequency characteristic. This method increases the dimensionality of the input, allowing it to acquire more information at the corresponding scale during the decoding process.

[0048] The low-frequency part of the aforementioned decoder is decoded using a vocoder, and this low-frequency part consists of an upsampling layer and a residual network.

[0049] The reason for setting separate low-frequency and high-frequency parameters in the aforementioned decoder is that while decoding using the inverse short-time Fourier transform (SFT) can significantly reduce the computational complexity, it places higher demands on model training in the low-frequency range, indirectly increasing the model's performance and real-time capabilities. Conversely, the high-frequency range has lower training requirements. Considering the application scenario of this embodiment, by using the inverse short-time Fourier transform for the high-frequency range and an upsampling layer for the low-frequency range, the computational resources required by the model can be significantly improved while maintaining the model's lightweight design.

[0050] The aforementioned lightweight speech conversion model significantly reduces the model's parameters and corresponding computational complexity, achieving both lightweight design and real-time performance to ensure a smooth user experience during calls. Furthermore, by using PPG features as constraints, the model addresses the instability introduced by simplification, enabling it to achieve good speech conversion results while maintaining real-time performance.

[0051] It should be noted that, for an audio clip, during processing, some features are explicitly known to indicate the timbre and rhythm of the audio; these features are referred to as audio features. Other features are explicitly known to indicate personal style; these features are referred to as style features. However, some features are unknown to the indicated object before training; their potential representation of timbre, rhythm, personal style, etc., can only be inferred from the model training results. These features are collectively defined as implicit features here.

[0052] It should be noted that the main method for extracting audio features in related technologies is through large-scale self-supervised models such as SoftHubert, Hubert, and WAV2Vec2.0. While this can yield richer features and knowledge, from the perspective of the final voice transformation result, the timbre of the target speech cannot accurately reflect the timbre characteristics or individuality of the target speaker's voice; the target speaker could even be a cartoon character. Specifically, because the aforementioned audio feature extraction methods do not specifically address the timbre characteristics of the target speaker, if the target speaker's timbre is extreme, such as that of certain singers with distinctive voices, the above audio feature extraction methods will lead to a certain degree of "averaging" of the target speaker's timbre during training. This means the target speaker's timbre will become similar to that of other speakers, thus failing to fully represent the target speaker's timbre. Consequently, the final voice transformation result will not achieve the desired effect. Therefore, the method of directly extracting audio features through large-scale self-supervised models in related technologies still has certain shortcomings in style transformation.

[0053] To address the aforementioned shortcomings, this application embodiment uses a pre-trained clustering model to perform clustering processing during audio feature extraction in the audio feature encoding module. By clustering the features, the quality of the features and the effect of sound conversion are improved, as detailed below.

[0054] The first step is to pre-set two clustering models, namely the first clustering model and the second clustering model described above. These two clustering models can be k-means models, but are not limited to them (K-means clustering, hierarchical clustering (SPSS), and DBSCAN clustering (MATLAB) are all acceptable). Both the first and second clustering models include feature extractors for feature extraction during the training process. The feature extractors for the first and second clustering models can be any two of the three types of models mentioned above: softhubert, hubert, and wav2vec2.0. For example, the first clustering model can use hubert as its feature extractor, and the second clustering model can use wav2vec2.0. The first and second clustering models can use different feature extractors.

[0055] The second step involves pre-training the first and second clustering models using general training data. This general training data can be based on LibriSpeech-960 and AISHELL-3 datasets, specifically acquiring speech samples from 200 speakers for each model, resulting in 200 clusters. The purpose of this training is to enable the two clustering models to classify speech samples from different speakers. Each model employs a different feature extractor, allowing for clustering from different dimensions. Different feature extraction methods have advantages in timbre, prosody, and speech rhythm; therefore, combining these different feature extraction methods can enrich the subsequent training and inference processes of the voice transformation system.

[0056] During the training of the clustering model, the cluster categories need to be manually encoded. For example, after the first clustering model clusters the speech sample data and obtains different categories, they can be assigned ID1.1, ID1.2, ..., ID1.9, and so on. Similarly, after the second clustering model clusters the speech sample data and obtains different categories, they can be assigned ID2.1, ID2.2, ..., ID2.9, and so on. Here, the purpose of category encoding is to give each category after clustering by the clustering model a unique identifier for differentiation, so as to facilitate category mapping and encoding during the subsequent training of the voice transformation system.

[0057] It should be noted that the clustering model involved in this application embodiment is trained in an unsupervised manner, that is, it is clustered solely based on the similarity of samples without relying on labels. The first training samples themselves do not have category labels. After clustering, category labels are manually assigned to sample sets of different types. Furthermore, the same category label does not necessarily correspond to the same speaker, but rather to speech with similar styles. For example, the first training sample includes 100 speech samples from different speakers. Through the clustering model, these can be divided into 10 sample sets, each corresponding to a certain style type, such as a hoarse male voice, a clear female voice, etc., and then these 10 sample sets are manually assigned corresponding category labels.

[0058] It should be noted that the aforementioned first and second clustering models do not participate in the construction of the audio feature encoding module; they only provide category encodings during the training phase of the audio feature encoding module. Similarly, for the trained audio feature encoding module, the first and second clustering models do not participate in the inference work during the actual sound conversion process. Correspondingly, these clustering models also do not participate in the subsequent inference part.

[0059] Figure 3This is a schematic diagram of an optional audio feature encoding module according to an embodiment of this application. It should be noted that the audio feature encoding module in this embodiment may include the following three units, such as... Figure 3 As shown:

[0060] Backbone network: Based on the Hubert network structure, it is used to extract and encode audio features.

[0061] Category mapping unit (linear projection): Consists of a mapping layer. This category mapping unit is used for category coding, that is, the category code corresponding to the audio feature map extracted by the feature coding unit.

[0062] Speech units: These consist of an embedding layer. During the training of the audio feature coding module, the speech units assign the category codes of the aforementioned clustering model to the audio features extracted by the feature coding units.

[0063] During training, the feature encoding units and category mapping units are first initialized, that is, some parameters of the Hubert network and mapping layers are randomly initialized. After initialization, the model is still trained using general training data. The general model data used for training here can be the same as or different from the training samples of the aforementioned clustering model; there are no restrictions on this.

[0064] Figure 4 This is a schematic diagram illustrating the training process of an optional audio feature encoding module according to an embodiment of this application, as shown below. Figure 4 As shown, in one embodiment, training the initial audio feature encoding module to convergence based on the true category labels output by the first clustering model and the second clustering model and the predicted category labels output by the initial audio feature encoding module includes:

[0065] The feature encoding unit and category mapping unit of the initial audio feature encoding module are initialized.

[0066] The feature encoding unit extracts and encodes the audio features of the third training sample, and the category mapping unit maps the audio features extracted by the feature encoding unit to the corresponding category code to obtain the predicted category label corresponding to the third training sample.

[0067] The predicted category label is output by the category encoding unit of the initial audio feature encoding module. Based on the predicted category label and the true category label output by the first clustering model and the second clustering model, the initial audio feature encoding module is trained until convergence.

[0068] In one embodiment, training the initial audio feature encoding module to convergence based on the predicted category labels and the true category labels output by the first clustering model and the second clustering model includes:

[0069] The loss function of the initial audio feature encoding module is updated based on the minimized first average cross-entropy between the true class label and the predicted class label output by the first clustering model, and the training parameters of the initial audio encoding module are also updated.

[0070] It should be noted that, during the training of the audio feature encoding module, in addition to the normal training and parameter updates of the feature encoding units, this embodiment also trains the audio encoding module based on the predicted class encoding of the corresponding class encoding of the training speech samples and the true class encoding ID obtained from the aforementioned clustering model. Specifically, the average cross-entropy between the true class encoding of Hubert's k-means (first clustering model) and the predicted class encoding of the audio encoding module is minimized, while the average cross-entropy between the true class encoding of wav2vec's k-means (second clustering model) and the predicted class encoding of the audio encoding module is also minimized. Based on this, the aforementioned loss is updated, and the parameters of the audio encoding module are updated accordingly. Through the above training method, the ability of the audio encoding module to classify timbre categories is further enhanced.

[0071] To address the issue that the style transfer effect of directly extracting audio features using large-scale self-supervised models in related technologies still has certain shortcomings, one embodiment of this application is based on a pre-trained clustering model. During the audio feature extraction process in the audio feature encoding module, clustering is performed on the features. By clustering the features, the extraction effect of personalized styles of some target speakers with strong personalities can be significantly improved, thereby improving the effect of subsequent voice conversion.

[0072] Since the clustering of features used in the aforementioned embodiments enables the extracted features for the target speaker to better represent the speaker's voice individuality, in order to further utilize the voice characteristics of the target speaker highlighted by the above features in the subsequent voice conversion process, this application embodiment further improves the timbre conversion part in the subsequent voice conversion process, so that it can better use style features to enhance the timbre conversion effect.

[0073] It should be noted that in the embodiments of this application, the scheme for style features in the timbre conversion process is based on the aforementioned clustering of features. That is, only through the clustered features can the subsequent timbre conversion be further realized through the timbre conversion module.

[0074] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described sound information protection method is also provided. This electronic device can be, but is not limited to, being used in a server. For example... Figure 5 As shown, the electronic device includes a memory 502 and a processor 504. The memory 502 stores a computer program, and the processor 504 is configured to execute the steps in any of the above method embodiments via the computer program.

[0075] Optionally, in this embodiment, the electronic device may be located in at least one of a plurality of network devices in a computer network.

[0076] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0077] S1, determine whether the current voice call belongs to the target call that needs to be converted; wherein, the target call includes at least: tagged harassing calls, sales calls and customer service calls, as well as calls from the caller that are not recorded in the current terminal's address book;

[0078] S2, after determining that the current voice call belongs to the target call and obtaining the instruction to perform voice conversion, the user's voice is converted into the target speaker's voice through a lightweight voice conversion model and then the user communicates with the current caller; wherein, the target speaker is a random third party or is specified by the current user, and the lightweight voice conversion model is a model trained using sample voice containing the target speaker's voice, including: a posterior encoder, a prior encoder and a decoder.

[0079] Alternatively, as those skilled in the art will understand, Figure 5 The structure shown is for illustrative purposes only. The electronic device can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile internet device (MID), a PAD, and other terminal devices. Figure 5 This does not limit the structure of the aforementioned electronic device. For example, the electronic device may also include components that are more... Figure 5 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 5 The different configurations shown.

[0080] The memory 502 can be used to store software programs and modules, such as the program instructions / modules corresponding to the sound information protection method and apparatus in this embodiment. The processor 504 executes various functional applications and data processing by running the software programs and modules stored in the memory 502, thereby implementing the aforementioned sound information protection method. The memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 502 may further include memory remotely located relative to the processor 504, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 502 may be used, but is not limited to, to store the program steps of the sound information protection method. As an example, such as... Figure 5 As shown, the memory 502 may include, but is not limited to, various modules from the lightweight speech conversion model described above. Furthermore, it may include, but is not limited to, other module units from the speech recognition neural network model described above, which will not be elaborated upon in this example.

[0081] Optionally, the transmission device 506 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 506 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 506 is a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0082] In addition, the aforementioned electronic device also includes: a display 508 for displaying input and output information for voice conversion; and a connection bus 510 for connecting various module components in the aforementioned electronic device.

[0083] Embodiments of this application also provide a computer-readable storage medium storing a computer program configured to execute the steps in any of the above method embodiments when run.

[0084] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0085] S1, determine whether the current voice call belongs to the target call that needs to be converted; wherein, the target call includes at least: tagged harassing calls, sales calls and customer service calls, as well as calls from the caller that are not recorded in the current terminal's address book;

[0086] S2, after determining that the current voice call belongs to the target call and obtaining the instruction to perform voice conversion, the user's voice is converted into the target speaker's voice through a lightweight voice conversion model and then the user communicates with the current caller; wherein, the target speaker is a random third party or is specified by the current user, and the lightweight voice conversion model is a model trained using sample voice containing the target speaker's voice, including: a posterior encoder, a prior encoder and a decoder.

[0087] Optionally, the storage medium is also configured to store a computer program for performing the steps included in the methods of the above embodiments, which will not be described again in this embodiment.

[0088] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0089] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0090] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0091] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0092] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0093] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0095] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for protecting sound information, characterized in that, include: Determine whether the current incoming voice call belongs to the target call that needs to be converted; wherein, the target call includes at least: tagged harassing calls, sales calls and customer service calls, as well as calls from the caller that are not recorded in the current terminal's address book; After determining that the current incoming voice call belongs to the target call and obtaining the instruction to perform voice conversion, the user's voice is converted into the target speaker's voice using a lightweight voice conversion model, and then the user can communicate with the current caller. The target speaker is a random third party or is specified by the current user. The lightweight voice conversion model is a model trained using sample voices containing the target speaker's voice, and includes a posterior encoder, a prior encoder, and a decoder. Before engaging in a conversation with the current caller after converting the current user's voice into the target speaker's voice using a lightweight speech conversion model, the method further includes: The audio feature encoding module identifies the style category of the target speaker's speech sample and extracts the audio features of the target speaker's speech sample based on the style category; wherein, the audio features of the target speaker's speech sample include the text features, prosodic features and timbre features of the target speaker's speech sample; The audio feature encoding module extracts the audio features corresponding to the current user's speech, wherein the audio features corresponding to the current user's speech include the text features, prosodic features and timbre features of the current user's speech; Before extracting the audio features corresponding to the current user's speech through the audio feature encoding module, the method further includes: A first clustering model is trained using a first training sample, wherein the first training sample includes speech samples from multiple speakers, and the speech samples from multiple speakers correspond to different style types; the first clustering model is used to cluster the first training sample and determine the category label corresponding to the first training sample based on the clustering result. A second clustering model is trained using a second training sample, wherein the second training sample includes speech samples from multiple speakers, and the speech samples from multiple speakers correspond to different style types; the second clustering model is used to cluster the second training sample and determine the category label corresponding to the second training sample based on the clustering result; the first clustering model and the second clustering model use feature extractors with different structures to perform clustering from different dimensions; The third training sample is simultaneously input into the first clustering model after training, the second clustering model after training, and the initial audio feature encoding module, wherein the third training sample contains speech samples from multiple speakers. The initial audio feature encoding module is trained according to the loss function of the initial audio feature encoding module. At the same time, the initial audio feature encoding module is trained until convergence according to the true category labels output by the first clustering model and the second clustering model and the predicted category labels output by the initial audio feature encoding module, thereby obtaining the audio feature encoding module. The audio feature encoding module is used to extract audio features according to the style type of speech.

2. The method as described in claim 1, characterized in that, The process of converting the current user's voice into the target speaker's voice using a lightweight speech conversion model and then communicating with the current caller includes: The posterior encoder fuses the audio features of the current user's speech with the audio features of the target speaker's speech sample to obtain joint encoded features, and extracts the first implicit features based on the audio features of the target speaker's speech sample. The prior encoder obtains the prior distribution of the first implicit feature from the phonemes of the current user's speech using a linear attention mechanism based on the posterior probability map (PPG) features of the target speaker's speech sample. The prior distribution is characterized by the mean or variance of the first implicit feature obtained with the PPG features as constraints. The decoder performs a standard streaming operation on the joint encoded features based on the prior distribution output by the prior encoder, and then outputs the converted speech through a vocoder. The decoder includes a high-frequency decoding module and a low-frequency decoding module. The high-frequency decoding module performs decoding using an inverse short-time Fourier transform, and the low-frequency decoding module performs decoding using a vocoder.

3. The method as described in claim 2, characterized in that, The step of obtaining the prior distribution of the first implicit feature from the phonemes of the current user's speech through the prior encoder, based on the posterior probability map (PPG) features of the target speaker's speech sample, using a linear attention mechanism, includes: A speech recognition model independent of the prior encoder is pre-trained; wherein the training sample data of the speech recognition model does not include speech samples of the target speaker; The PPG features of the target speaker's speech sample are extracted using the trained speech recognition model; The constraint loss during the constraint process is calculated by using a PPG feature prediction unit that is independent of the prior encoder, and the extracted PPG features are compensated for.

4. The method as described in claim 2, characterized in that, After obtaining the prior distribution of the first implicit feature from the phonemes of the current user's speech through the prior encoder based on the posterior probability map (PPG) features of the target speaker's speech sample using a linear attention mechanism, the method further includes: The prior distribution of the first implicit feature is complicated by the normalization layer in the prior encoder, and the corresponding spectral information is output. The normalization layer is composed of multiple coupling layers, each of which is composed of multiple residual blocks, and the multiple coupling layers share parameters.

5. The method as described in claim 2, characterized in that, The lightweight speech conversion model includes the following steps during training: Obtain the audio features of the target speaker's speech sample and its corresponding second implicit features; A predicted value of the target speaker's speech sample is generated based on the audio features of the target speaker's speech sample and the second implicit feature; A first loss function is trained based on the predicted value of the target speaker's speech sample and the actual value of the target speaker's speech sample; wherein the first loss function is used to indicate the loss of the predicted value of the target speaker's speech sample relative to the actual value; The alignment prediction unit aligns the audio features of the target speaker's speech sample with the second implicit features to calculate the alignment matrix, and trains a second loss function based on the alignment matrix. The second loss function is used to indicate the dispersion between the audio features of the target speaker's speech sample and the second implicit features. The identification result is determined based on the difference between the predicted value and the true value of the target speaker's speech sample, and a third loss function is trained based on the identification result; The trained lightweight speech conversion model is obtained based on the first loss function, the second loss function, and the third loss function.

6. A sound information protection device, said device being applicable to the method described in any one of claims 1 to 5, characterized in that, include: The determination module is configured to determine whether the current incoming voice call belongs to the target call that needs to be converted; wherein, the target call includes at least: tagged harassing calls, sales calls and customer service calls, as well as calls from the caller that are not recorded in the current terminal's address book; The conversion module is configured to, after determining that the current incoming voice call belongs to the target call and obtaining an instruction to perform voice conversion, convert the current user's voice into the voice of the target speaker using a lightweight voice conversion model and then conduct a call with the current caller; wherein, the target speaker is a random third party or is specified by the current user, and the lightweight voice conversion model is a model trained using sample voice containing the voice of the target speaker, including: a posterior encoder, a prior encoder, and a decoder.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 5 when it is run.

8. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as described in any one of claims 1 to 5.

Images