Voice processing method, electronic device, storage medium and program product

Abstract

The invention provides a voice processing method, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a first voice; extracting a first feature and at least one second feature from the first voice; acquiring a third feature of the same category as the first feature; and generating a second voice based on the third feature and the at least one second feature. According to the voice processing method, the electronic equipment, the storage medium and the program product, only the voice features corresponding to the first feature and the third feature can be desensitized in the audio desensitization process, and other voice features are completely reserved, so that the data availability is protected while the personal information security of a user is protected, and the user experience is improved. And the tone desensitization effect is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a voice processing method, an electronic device, a storage medium and a program product. BACKGROUND

[0002] With the rapid development of mobile internet technology and audio and video technology, voice has become one of the most natural and efficient ways of human-computer interaction. From intelligent sound boxes, voice assistants to online meetings, all of them are based on voice interaction to realize business services.

[0003] Voice timbre desensitization technology refers to modifying voice timbre features that can represent the identity of a speaker, so that the original speaker's identity cannot be identified through voice recognition, thereby protecting personal privacy information.

[0004] However, the existing voice timbre desensitization method not only eliminates the voice timbre features, but also eliminates other voice features, which reduces the usability of voice in business services. SUMMARY

[0005] Therefore, the present disclosure aims to provide a voice processing method, an electronic device, a storage medium and a program product.

[0006] To achieve the above purpose, the first aspect of the present disclosure provides a voice processing method, comprising: obtaining a first voice; extracting a first feature and at least one second feature from the first voice; obtaining a third feature of the same category as the first feature; generating a second voice based on the third feature and the at least one second feature.

[0007] In some embodiments, the first feature includes a first voice timbre feature for representing the voice timbre of the first voice, and the third feature includes a second voice timbre feature different from the first voice timbre feature.

[0008] In some embodiments, the third feature of the same category as the first feature is obtained by: obtaining a voice timbre feature with a similarity to the first voice timbre feature satisfying a preset threshold range from a preset voice timbre feature set, and determining the second voice timbre feature based on the voice timbre feature.

[0009] In some embodiments, the at least one second feature includes an audio feature and a fundamental frequency feature, and the second voice is generated based on the third feature and the at least one second feature, comprising: performing feature fusion on the audio feature and the second voice timbre feature to obtain a first fusion feature; The first fused feature and the fundamental frequency feature are fused to obtain the second fused feature; The second speech is generated based on the second fusion feature.

[0010] In some embodiments, the feature fusion of the audio feature and the second timbre feature to obtain a first fused feature includes: Obtain the first statistical feature of the audio feature in the time dimension and the second statistical feature of the second timbre feature in the time dimension; The first statistical feature is normalized based on the second statistical feature to obtain the first fused feature.

[0011] In some embodiments, fusing the first fused feature with the fundamental frequency feature to obtain the second fused feature includes: Obtain the third statistical feature of the first fusion feature in the time dimension and the fourth statistical feature of the fundamental frequency feature in the time dimension; The third statistical feature is normalized based on the fourth statistical feature to obtain the second fusion feature.

[0012] In some embodiments, wherein: Input the first speech into the first model, and obtain the second speech output by the first model; The first model includes: a feature extraction module, a feature processing module, and a speech generation module; The feature extraction module is used to extract the first feature and the at least one second feature from the first speech; The feature processing module is used to output the third feature, which belongs to the same category as the first feature; The speech generation module is used to generate the second speech based on the third feature and at least one second feature.

[0013] In some embodiments, the speech generation module includes a fusion module and an audio reconstruction module: The fusion module is used to fuse the third feature and the at least one second feature to obtain a third fused feature; The audio reconstruction module is used to reconstruct the audio from the third fusion feature to generate the second speech.

[0014] In some embodiments, the training process of the first model includes: Obtain a test set for the first model, the test set including multiple test samples, each test sample including a third speech; The third speech is input into the first model to obtain the fourth speech; wherein, the first model includes the feature extraction module and the speech generation module, the feature extraction module is used to extract a fourth feature and at least one fifth feature from the third speech, and the speech generation module generates the fourth speech based on the fourth feature and the at least one fifth feature; Obtain the loss information between the fourth speech and the third speech, perform backpropagation based on the loss information, and update the parameters of the first model.

[0015] In some embodiments, the fourth feature includes a third timbre feature for characterizing the timbre of the third speech.

[0016] In some embodiments, the speech generation module includes a fusion module and an audio reconstruction module; The fusion module is used to fuse the fourth feature and at least one fifth feature to obtain a fourth fused feature; The audio reconstruction module is used to reconstruct the audio from the fourth fusion feature and generate the fourth speech.

[0017] In some embodiments, obtaining the first loss information between the fourth speech and the third speech includes: The loss information is calculated based on the loss function; The loss function includes the sum of reconstruction loss and generator loss. The reconstruction loss is obtained based on the average absolute error between the fourth speech and the third speech, and the generator loss is obtained based on the loss during the audio reconstruction process.

[0018] A second aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the voice processing method as described in the first aspect.

[0019] A third aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the speech processing method described in the first aspect.

[0020] This disclosure provides a fourth aspect of a computer program product, including computer program instructions that, when executed on a computer, cause the computer to perform the speech processing method as described in the first aspect.

[0021] As can be seen from the above, the speech processing method, electronic device, storage medium, and program product provided in this disclosure process the first speech. During the processing, the first feature extracted from the first speech is replaced by a third feature of the same type as the first feature generated by the feature processing module, thereby obtaining the second speech. The difference between the second speech and the first speech is only the speech information corresponding to the first and third features, while other speech information remains unchanged. Thus, in the audio desensitization process, only the speech features corresponding to the first and third features are desensitized, while other speech features are fully preserved. This protects the security of users' personal information while also protecting data availability and improving the effectiveness of audio desensitization. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this disclosure or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram of an exemplary system provided by an embodiment of this disclosure is shown.

[0024] Figure 2 A flowchart illustrating an exemplary method provided by an embodiment of this disclosure is shown.

[0025] Figure 3 A schematic flowchart of an exemplary model reasoning process according to an embodiment of this disclosure is shown.

[0026] Figure 4 Another schematic diagram of exemplary model reasoning according to an embodiment of this disclosure is shown.

[0027] Figure 5 A schematic diagram of an exemplary model training process according to an embodiment of this disclosure is shown.

[0028] Figure 6 A schematic diagram of the hardware structure of an exemplary computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0030] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0031] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner, and user authorization will be obtained.

[0032] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations of this disclosed technical solution.

[0033] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0034] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0035] As described in the background section, with the rapid development of mobile internet and audio / video technologies, voice has become one of the most natural and efficient ways of human-computer interaction. From smart speakers and voice assistants to online meetings and customer service centers, users' voice data is being collected and applied more extensively than ever before. Voice not only carries the content information that the speaker wants to convey, but also contains the speaker's timbre characteristics. Timbre characteristics are unique and difficult to forge biometric features, which can be used to trace the speaker.

[0036] Voice timbre, often likened to a "fingerprint of the voice," is a key attribute that distinguishes the voices of different individuals. It allows users to easily identify the voices of friends, family, and even public figures in a crowd. However, precisely because of this, the leakage or misuse of raw voice data can pose a serious threat to personal privacy and security.

[0037] Voice anonymization technology is a technique that anonymizes the speaker's voice in speech, protecting personal privacy in speech information from a voice perspective. While preserving as much of the original content, rhythm, and emotion as possible, voice anonymization technology uses technical means to desensitize the speech, modifying the voice features that identify the speaker, making it difficult for human listeners or artificial intelligence systems to easily identify the original speaker, thus decoupling the speech from the original speaker's identity.

[0038] Voice anonymization technology has a wide range of applications in privacy protection. For example, in the development of speech-related artificial intelligence algorithms, especially in model training scenarios, there is often a need for massive amounts of speech data. To protect the security of users' personal information, it is necessary to anonymize the collected data. Voice anonymization technology can "clean" this data, making it untraceable to the specific speaker while retaining its training value.

[0039] However, existing tone desensitization technologies have the following problems: (1) Desensitize voice information to protect privacy while also protecting data availability. The deeper the degree of voice desensitization, the better the privacy protection effect, but the auditory effect of the audio will also be affected.

[0040] (2) Timbre is often intertwined with paralinguistic information such as emotion, accent, speaking style, stress and intention. Existing technologies cannot desensitize timbre features while fully preserving paralinguistic information such as the speaker's emotion, accent, speaking style, stress and intention.

[0041] In view of this, embodiments of this disclosure provide a speech processing method that acquires a first speech, extracts a first feature and at least one second feature from the first speech, acquires a third feature of the same category as the first feature, and generates a second speech based on the third feature and the at least one second feature. This method anonymizes the first feature in the speech and ensures that only the first feature changes in the de-identified speech, while other features remain unchanged, thus guaranteeing the auditory effect of the speech and maintaining clarity and naturalness of the language content. Simultaneously, while eliminating identity information, it avoids mistakenly eliminating other features of the speaker, thereby solving the privacy and usability issues of traditional technologies.

[0042] After introducing the basic principles of this disclosure, various non-limiting embodiments of this disclosure will be described in detail below.

[0043] Figure 1 A schematic diagram of an exemplary system 100 provided in an embodiment of this disclosure is shown.

[0044] like Figure 1 As shown, system 100 can be used to implement online or offline voice processing functions and may include terminal devices 102A and 102B, server 106, and database server 108. A medium (e.g., a network) may be included to provide a communication link between terminal devices 102A and 102B and server 106 and database server 108. This network may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0045] The terminal devices 102A and 102B can be equipped with various applications (APPs) or software, such as voice processing software, or project management applications or software, collaborative office applications or software, image processing applications or software, video conferencing applications or software, reading applications or software, video applications or software, social applications or software, payment applications or software, web browsers, and instant messaging tools that include audio processing functions. In some embodiments, these applications or software can all be used for voice processing.

[0046] The terminal devices 102A and 102B here can be either hardware or software. When terminal devices 102A and 102B are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, e-book readers, MP3 players, laptops, and desktop computers (PCs). When terminal devices 102A and 102B are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module. No specific limitations are set here.

[0047] Server 106 can be a server that provides various services, such as a backend server that supports various applications displayed on terminal devices 102A and 102B. Database server 108 can also be a database server that provides various services. It is understood that if server 106 can implement the relevant functions of database server 108, database server 108 may not need to be set up in system 100.

[0048] The server 106 and database server 108 here can be either hardware or software. When they are hardware, they can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When they are software, they can be implemented as multiple software programs or software modules (for example, to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0049] It should be noted that the voice processing method provided in this embodiment can be executed by server 106, or by terminal devices 102A and 102B. It should be understood that... Figure 1 The number of terminal devices, users, servers, and database servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, users, servers, and database servers.

[0050] In some embodiments, terminal devices 102A and 102B may have voice processing applications or software installed. User 104A may use the application or software installed in terminal device 102A to train the first model, and user 104B may use the application or software installed in terminal device 102B to perform inference on the first model.

[0051] The following is combined with Figure 1 The above application scenarios are used to describe the speech processing method according to exemplary embodiments of this disclosure. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this disclosure, and the embodiments of this disclosure are not limited in any way. Rather, the embodiments of this disclosure can be applied to any applicable scenario.

[0052] Reference Figure 2 As shown, the speech processing method includes: Step S101: Obtain the first audio recording.

[0053] In this embodiment, since the first speech needs to be desensitized by timbre, the first speech needs to include human voice.

[0054] The first speech can be speech information that includes only human voice, or it can be speech information obtained from audio that includes multiple types of information such as human voice and music. This embodiment does not limit this.

[0055] The first speech may include the voice of one speaker or the voices of multiple speakers; this embodiment does not limit this.

[0056] Step S103: Extract a first feature and at least one second feature from the first speech.

[0057] The first feature can be a timbre feature, thereby enabling desensitization of the timbre in the first speech based on the first feature. When the first feature is a timbre feature, at least one second feature is another feature in the first speech besides the timbre feature, such as audio features, fundamental frequency features, prosodic features, timbre features, vocalization type, etc., which are not limited in this embodiment.

[0058] Alternatively, the first feature can also be an audio feature, fundamental frequency feature, prosodic feature, phonological feature, or vocalization type, thereby enabling desensitization of corresponding features in the first speech based on the first feature. When the first feature is an audio feature, fundamental frequency feature, prosodic feature, phonological feature, or vocalization type, at least one second feature is a feature in the first speech other than the first feature; this embodiment does not limit this.

[0059] Step S105: Obtain a third feature of the same category as the first feature.

[0060] Among them, the third feature is of the same type as the first feature. That is, when the first feature is a timbre feature, then the third feature is also a timbre feature, and the timbre of the third feature is not the same as that of the first feature. When the first feature is another type of feature such as an audio feature, fundamental frequency feature, prosody feature, timbre feature, or vocal type, then the third feature is also another type of feature such as an audio feature, fundamental frequency feature, prosody feature, timbre feature, or vocal type, but the specific feature values ​​of the third feature and the first feature are not the same.

[0061] Step S107: Generate a second speech based on the third feature and the at least one second feature.

[0062] In this embodiment, a second speech can be generated based on a third feature and at least one second feature. For example, feature fusion can be performed on the third feature and at least one second feature, and audio reconstruction can be performed on the feature fusion result to obtain the second speech.

[0063] In this embodiment, the first speech is processed. During the processing, the first feature extracted from the first speech is replaced by a third feature of the same type as the first feature, thereby obtaining the second speech. The difference between the second speech and the first speech is only the speech information corresponding to the first and third features, while other speech information remains unchanged. Thus, in the audio desensitization process, only the speech features corresponding to the first and third features are desensitized, while other speech features are fully preserved. This protects the security of users' personal information while also protecting data availability and improving the effectiveness of audio desensitization.

[0064] In some embodiments, the first feature includes a first timbre feature for characterizing the timbre of the first speech, and the third feature includes a second timbre feature that is different from the first timbre feature.

[0065] In this embodiment, the first feature can be a first timbre feature, and the third feature is a second timbre feature that is different from the first timbre feature.

[0066] The first timbre feature is used to characterize the timbre of the human voice in the first speech, and this first timbre feature can be used to identify the speaker's identity.

[0067] In this embodiment, after extracting the first timbre feature and at least one second feature of the first speech, the first timbre feature is first replaced with the second timbre feature, wherein the second timbre feature is different from the timbre represented by the first timbre feature; then the second speech is generated based on the second timbre feature and at least one second feature.

[0068] In this process, only the timbre feature is replaced. That is, the difference between the second and first speech is only in timbre. Other features, such as paralinguistic features including the speaker's emotion, accent, speaking style, stress, and intention, as well as the fundamental frequency feature including the pitch contour of the speaker's voice, remain unchanged. Thus, in the audio desensitization process, only the timbre feature is desensitized, while the paralinguistic information such as the speaker's emotion, accent, speaking style, stress, and intention is fully preserved. This protects the user's personal information security while also protecting data availability and improving the effectiveness of timbre desensitization.

[0069] In this embodiment, the first speech is processed to obtain the second speech. The second speech differs from the first speech only in timbre. Other features, such as the speaker's emotions, accent, speaking style, stress, and intention, as well as the fundamental frequency features of the speaker's voice pitch profile, remain unchanged. Thus, in the audio desensitization process, only the timbre features are desensitized, while the speaker's emotions, accent, speaking style, stress, and intention are fully preserved. This protects the user's personal information security while also protecting data availability and improving the effectiveness of timbre desensitization.

[0070] In some embodiments, the method further includes: obtaining timbre features from a preset set of timbre features whose similarity to the first timbre feature meets a preset threshold range, and determining the second timbre feature based on the timbre feature.

[0071] In this embodiment, a pre-set set of timbre features can be obtained. The similarity between each timbre feature in the set and the first timbre feature is calculated, and any timbre feature whose similarity satisfies a preset threshold range is selected as the second timbre feature. Alternatively, the timbre feature with the highest similarity value within the preset threshold range can be selected as the second timbre feature. By replacing the first timbre feature with the second timbre feature that is similar to it, the paralinguistic information in the first speech can be preserved as much as possible during desensitization. In this embodiment, the number of first timbre features can be one or more; when the first speech includes multiple speakers, there can be multiple first timbre features, for example, the same number as the number of speakers.

[0072] When there are multiple first timbre features, there are also multiple second timbre features, and the number of second timbre features is the same as that of first timbre features. In this way, when the timbre of multiple speakers in the first speech is replaced, the timbre after replacement of different speakers will not be the same, thereby improving the usability of audio data after timbre desensitization.

[0073] In some embodiments, the at least one second feature includes an audio feature and a fundamental frequency feature, and generating a second speech based on the third feature and the at least one second feature includes: fusing the audio feature, the fundamental frequency feature and the second timbre feature to obtain a second fused feature.

[0074] Audio features are used to characterize the language content of the human voice in the first speech and the paralinguistic features other than the first timbre feature. The language content is what the speaker said, and the paralinguistic features other than the first timbre feature can include the speaker's emotion, accent, speaking style, stress and intention, etc.

[0075] Fundamental frequency features are used to characterize the pitch features of human voice in the first speech, that is, the pitch profile of the speaker's voice. Fundamental frequency features can represent the natural rhythm and intonation of the original speech.

[0076] In this embodiment, audio features in the first speech can be extracted by an audio encoder. These audio features include all important features in the first speech except for timbre features, including language content and paralinguistic features such as the speaker's emotion, accent, speaking style, stress, and intention.

[0077] In some embodiments, the audio encoder can be a pre-trained speech model that can recognize not only language content but also paralinguistic features.

[0078] In this embodiment, the fundamental frequency (F0) features of the first speech can be extracted using a fundamental frequency extractor, which is the pitch profile of the sound, i.e., the pitch emitted at each moment. Based on the fundamental frequency features, the natural rhythm and intonation of the speech can be determined.

[0079] In some embodiments, the fundamental frequency extractor can be a preset pitch estimation model, which can be based on a graphics processing unit (GPU) to extract fundamental frequency features, thereby greatly improving processing speed and enabling real-time timbre desensitization.

[0080] In this embodiment, a first timbre feature of the first speech can be extracted by a speaker encoder. This timbre feature is used to identify the identity of one or more speakers in the first speech.

[0081] In some embodiments, the speaker encoder can be a pre-trained model for a speaker recognition task, which is implemented based on a neural network architecture and trained on a real-world speaker recognition dataset. This pre-trained model can accurately identify the first timbre feature in the first speech.

[0082] In this embodiment, after obtaining the audio features, fundamental frequency features, and second timbre features, the audio features, fundamental frequency features, and second timbre features are fused to obtain the second fused feature.

[0083] In some embodiments, fusing the audio feature, the fundamental frequency feature, and the second timbre feature to obtain a second fused feature includes: first fusing any two features from the audio feature, the fundamental frequency feature, and the second timbre feature, and then fusing the fused feature with the unfused feature, for example: First, the audio features and fundamental frequency features are fused together. Then, the fused features of the two are fused together with the second timbre features to obtain the second fused features.

[0084] Alternatively, the fundamental frequency feature and the second timbre feature can be fused first, and then the fused feature of the two can be fused with the audio feature to obtain the second fused feature.

[0085] In some embodiments, generating the second speech based on the third feature and the at least one second feature includes: Step S201: The audio feature is fused with the second timbre feature to obtain the third fused feature.

[0086] In some embodiments, the feature fusion of the audio feature and the second timbre feature in step S201 to obtain the first fused feature includes: Step S2011: Obtain the first statistical feature of the audio feature in the time dimension and the second statistical feature of the second timbre feature in the time dimension.

[0087] In this embodiment, the mean and standard deviation of the audio features in the time dimension can be calculated to obtain the first statistical feature; the mean and standard deviation of the second timbre feature in the time dimension can be calculated to obtain the second statistical feature.

[0088] Step S2013: Normalize the first statistical feature based on the second statistical feature to obtain the third fusion feature.

[0089] In this embodiment, the second statistical feature of the second timbre feature is used to normalize the first statistical feature of the audio feature, thereby integrating the second timbre feature into the audio feature and obtaining the first fused feature.

[0090] Step S203: The first fused feature and the fundamental frequency feature are fused to obtain the second fused feature.

[0091] In some embodiments, the feature fusion of the first fused feature and the fundamental frequency feature in step S203 to obtain the second fused feature includes: Step S2031: Obtain the third statistical feature of the first fusion feature in the time dimension and the fourth statistical feature of the fundamental frequency feature in the time dimension.

[0092] In this embodiment, the mean and standard deviation of the first fusion feature in the time dimension can be calculated to obtain the third statistical feature; the mean and standard deviation of the fundamental frequency feature in the time dimension can be calculated to obtain the fourth statistical feature.

[0093] Step S2033: Normalize the third statistical feature based on the fourth statistical feature to obtain the second fusion feature.

[0094] In this embodiment, the third statistical feature of the third fusion feature is normalized using the fourth statistical feature of the fundamental frequency feature, thereby integrating the fundamental frequency feature into the third fusion feature to obtain the second fusion feature, thus realizing feature fusion.

[0095] Step S205: Generate the second speech based on the second fusion feature.

[0096] In this embodiment, the audio features and the second timbre features can be fused first, and then the fused features of the two can be fused with the fundamental frequency features to obtain the second fused features, and then the second speech can be generated based on the second fused features.

[0097] In this embodiment, since the correlation between the audio features and the second timbre features is higher, the audio features and the second timbre features are fused first, and then the fused features are fused with the fundamental frequency features, so as to obtain a better feature fusion effect.

[0098] In some embodiments, generating the second speech based on the second fusion feature includes: performing audio reconstruction on the second fusion feature to generate the second speech.

[0099] In some embodiments, a second speech can be obtained by generating an audio waveform signal with a time sequence based on a second fusion feature using a waveform generator.

[0100] In some embodiments, the method further includes: inputting the first speech into a first model and obtaining the second speech output by the first model; wherein the first model includes: a feature extraction module, a feature processing module, and a speech generation module; the feature extraction module is used to extract the first feature and the at least one second feature from the first speech; the feature processing module is used to output the third feature of the same category as the first feature; and the speech generation module is used to generate the second speech based on the third feature and the at least one second feature.

[0101] In this embodiment, the first model is trained to achieve timbre desensitization, so that after the first speech is input into the first model, the first model can directly output the second speech, wherein the timbre of the second speech is different from that of the human voice in the first model.

[0102] In some embodiments, such as Figure 3 As shown, the first model may include a feature extraction module, a feature processing module, and a speech generation module.

[0103] When the first speech is input into the first model, the feature extraction module can extract features from the input first speech to obtain a first feature and at least one second feature.

[0104] The feature processing module generates a third feature. This third feature is of the same type as the first feature. In some embodiments, both the third and first features are timbre features used to characterize timbre, and the third and first features are timbre features used to characterize different timbres.

[0105] The speech generation module generates a second audio based on at least one second feature extracted by the feature extraction module and a third feature generated by the feature processing module.

[0106] In this embodiment, after the first model extracts the first feature and at least one second feature of the first speech, it replaces the first feature with a third feature of the same type as the first feature, and then generates the second speech based on the third feature and at least one second feature.

[0107] In this process, only the first feature is replaced by a third feature of the same type as the first feature, while other features remain unchanged. Thus, in the audio desensitization process, only the voice information corresponding to the first feature is desensitized, while the speaker's other voice information is fully preserved. This protects the user's personal information security while also protecting data availability and improving the effectiveness of audio desensitization.

[0108] In this embodiment, the first speech is processed by the first model. During the processing, the first feature extracted from the first speech is replaced by a third feature of the same type generated by the feature processing module, thereby obtaining the second speech. The difference between the second speech and the first speech is only the speech information corresponding to the first and third features, while other speech information remains unchanged. Thus, in the audio desensitization process, only the speech features corresponding to the first and third features are desensitized, while other speech features are fully preserved. This protects the security of users' personal information while also protecting data availability and improving the effectiveness of audio desensitization.

[0109] In some embodiments, when the first feature includes a first timbre feature, at least one second feature includes an audio feature and a fundamental frequency feature, and the third feature includes a second timbre feature, after the first speech is input into the first model, the first timbre feature, audio feature, and fundamental frequency feature of the first speech can be extracted respectively through the feature extraction module of the first model.

[0110] like Figure 4 As shown, the feature extraction module includes an audio encoder, a baseband extractor, and a speaker encoder.

[0111] In some embodiments, the feature processing module may be, for example, an anonymizer, through which the second timbre feature is obtained.

[0112] In this embodiment, audio features in the first speech can be extracted by an audio encoder, fundamental frequency (F0) features of the first speech can be extracted by a fundamental frequency extractor, and first timbre features of the first speech can be extracted by a speaker encoder.

[0113] After the first model extracts the first timbre feature, audio feature and fundamental frequency feature of the first speech, it first replaces the first timbre feature with the second timbre feature, wherein the second timbre feature is different from the timbre represented by the first timbre feature; then it generates the second speech based on the second timbre feature, audio feature and fundamental frequency feature.

[0114] In this process, only the timbre feature is replaced. That is, the difference between the second and first speech is only in timbre. Other features, such as paralinguistic features including the speaker's emotion, accent, speaking style, stress, and intention, as well as the fundamental frequency feature including the pitch contour of the speaker's voice, remain unchanged. Thus, in the audio desensitization process, only the timbre feature is desensitized, while the paralinguistic information such as the speaker's emotion, accent, speaking style, stress, and intention is fully preserved. This protects the user's personal information security while also protecting data availability and improving the effectiveness of timbre desensitization.

[0115] In this embodiment, the first speech is processed by the first model to obtain the second speech. The second speech differs from the first speech only in timbre. Other features, such as the speaker's emotions, accent, speaking style, stress, and intentions, as well as the fundamental frequency features of the speaker's voice pitch profile, remain unchanged. Thus, in the audio desensitization process, only the timbre features are desensitized, while the speaker's emotions, accent, speaking style, stress, and intentions are fully preserved. This protects the user's personal information security while also protecting data availability and improving the effectiveness of timbre desensitization.

[0116] In some embodiments, since the audio data is processed through the first model and the audio data after timbre desensitization is directly output based on the first model, the speech processing method for speech feature desensitization described in this disclosure is implemented based on an end-to-end audio desensitization model architecture.

[0117] In some embodiments, the feature processing module obtains a second timbre feature from a preset timbre feature set whose similarity to the first timbre feature meets a preset threshold.

[0118] In some embodiments, the speech generation module performs speech generation on the audio features, the fundamental frequency features, and the second timbre features to obtain a second speech.

[0119] In this embodiment, after obtaining the audio features, fundamental frequency features, and second timbre features, speech is generated based on the audio features, fundamental frequency features, and second timbre features to obtain the second speech.

[0120] like Figure 4 As shown, the speech generation module includes a fusion module, which performs feature fusion on audio features, fundamental frequency features and second timbre features to obtain the second fused feature.

[0121] In some embodiments, the speech generation module further includes an audio reconstruction module, which is used to perform audio reconstruction on the second fusion feature to generate the second speech.

[0122] like Figure 4 As shown, the speech generation module also includes an audio reconstruction module. The fusion module fuses the third feature and at least one second feature to obtain a second fused feature. The audio reconstruction module then reconstructs the audio from the second fused feature output by the fusion module, thereby obtaining the second speech.

[0123] In some embodiments, the audio reconstruction module may include a waveform generator, which can generate an audio waveform signal with a time sequence based on a second fusion feature, thereby obtaining a second speech.

[0124] In some embodiments, the training process of the first model includes: Step S301: Obtain a test set for the first model, the test set including multiple test samples, each test sample including a third speech.

[0125] In this embodiment, a large amount of speech data containing one or more speakers is acquired, and audio segments are randomly selected as third speech to obtain a test set for the first model.

[0126] Step S303: Input the third speech into the first model to obtain the fourth speech; wherein, the first model has a feature extraction module and a speech generation module, the feature extraction module is used to extract a fourth feature and at least one fifth feature from the third speech, and the speech generation module is used to generate the fourth speech based on the fourth feature and the at least one fifth feature.

[0127] like Figure 3 As shown, in this embodiment, the third speech from the test set is input into the first model to generate the fourth speech.

[0128] During training, after the third speech is input into the first model, the feature extraction module of the first model first extracts a fourth feature and at least one fifth feature of the third speech. In some embodiments, the fourth feature is a third timbre feature used to characterize the timbre of the third speech, and the at least one fifth feature includes the audio features and fundamental frequency features of the third speech.

[0129] In some embodiments, such as Figure 3As shown, the speech generation module includes a fusion module and an audio reconstruction module; the fusion module is used to fuse the fourth feature and at least one fifth feature to obtain a fourth fused feature; the audio reconstruction module is used to reconstruct the audio from the fourth fused feature to generate the fourth speech.

[0130] like Figure 5 As shown, the feature extraction module of the first model includes an audio encoder, a fundamental frequency extractor, and a speaker encoder.

[0131] In this embodiment, audio features in the third speech can be extracted by an audio encoder, fundamental frequency (F0) features of the third speech can be extracted by a fundamental frequency extractor, and third timbre features of the third speech can be extracted by a speaker encoder.

[0132] After extracting the audio features, fundamental frequency features, and third timbre features of the third speech through the feature extraction module of the first model, the audio features, fundamental frequency features, and third timbre features of the third speech are then fused to obtain the fourth fused feature.

[0133] like Figure 5 As shown, the speech generation module includes a feature fusion module, which fuses audio features, fundamental frequency features, and third timbre features to obtain a fourth fused feature.

[0134] After the first model obtains the fourth fusion feature, it performs audio reconstruction to obtain the fourth speech.

[0135] like Figure 5 As shown, the speech generation module also includes an audio reconstruction module, which performs audio reconstruction on the fourth fusion feature to obtain the fourth speech.

[0136] In some embodiments, the audio reconstruction module may include a waveform generator that can generate an audio waveform signal with a time sequence based on a third fusion feature, thereby obtaining a fourth speech.

[0137] Step S305: Obtain the loss information between the fourth speech and the third speech, perform backpropagation based on the loss information, and update the parameters of the first model.

[0138] In this embodiment, after the fourth speech is generated, the loss information between the fourth speech and the third speech is calculated.

[0139] Ideally, the fourth speech generated by the first model would be exactly the same as the third speech, and the loss between the two would be zero.

[0140] However, in practice, losses may occur during feature extraction, feature fusion, and reconstruction. To reduce these losses, this embodiment calculates the loss information between the fourth and third speech segments, calls the backpropagation method of the loss information to calculate the gradient of the model parameters of the first model, and uses an optimizer to update the model parameters.

[0141] In some embodiments, obtaining the loss information between the fourth speech and the third speech in step S305 includes: calculating the loss information based on a loss function; wherein the loss function includes the sum of reconstruction loss and generator loss, the reconstruction loss is obtained based on the mean absolute error between the fourth speech and the third speech, and the generator loss is obtained based on the loss during the audio reconstruction process.

[0142] In this embodiment, the loss function L is as follows:

[0143] in, Indicates the losses incurred during reconstruction. This represents the generator loss.

[0144] In this embodiment, the reconstruction loss is used to measure the difference in loss between the original third speech and the generated fourth speech. For example, the reconstruction loss can be obtained by calculating the mean absolute error between the fourth and third speech. The mean absolute error between the fourth and third speech represents the average degree of difference between them at each time frame and each frequency.

[0145] In some embodiments, the L1 loss based on the Mel spectrum can be used as the loss function for reconstruction loss.

[0146] In this embodiment, the generator loss is used to make the timbre features of the fourth speech similar to those of the third speech, thereby making the generated fourth speech sound closer to machine-generated speech that resembles a human voice. In this embodiment, the generator loss can be obtained based on the adversarial loss, reconstruction loss, and feature matching loss in the audio reconstruction process.

[0147] In some embodiments, the audio reconstruction module may be based on an adversarial network model, which includes a discriminator and a generator. Specifically, the adversarial loss enables the fourth speech generated by the generator to deceive the discriminator, making the audio generated by the discriminator appear authentic; the reconstruction loss ensures that the generated fourth speech is content-wise consistent with the third speech; and the feature matching loss ensures that the fourth speech generated by the generator is similar to the third speech in the intermediate layer features of the discriminator.

[0148] In some embodiments, the weight of the reconstruction loss in the loss function L is 0.3 to 0.6, and the weight of the generator loss is 0.7 to 0.4.

[0149] In some embodiments, the reconstruction loss has a weight of 0.3 and the generator loss has a weight of 0.7; or, the reconstruction loss has a weight of 0.4 and the generator loss has a weight of 0.6; or, the reconstruction loss has a weight of 0.5 and the generator loss has a weight of 0.5; or, the reconstruction loss has a weight of 0.6 and the generator loss has a weight of 0.4. This embodiment does not limit the specific weight of the reconstruction loss.

[0150] The speech processing method described in this embodiment adopts an end-to-end timbre desensitization model architecture. In the first model, a feature decoupling and reconstruction paradigm is used. During training of the first model, an audio reconstruction paradigm is employed, using audio reconstruction loss to enable the model to decompose and reconstruct audio features. In timbre desensitization, timbre desensitization is achieved by replacing anonymous timbre features. The replaced audio retains information such as content, rhythm, and prosody from the original audio. Feature fusion ensures the merging effect of anonymous timbre features with other audio features, ultimately synthesizing the desensitized audio. Therefore, the speech processing method described in this embodiment for speech feature desensitization has excellent timbre desensitization technical effects, ensuring high privacy of the desensitized audio while also possessing high usability and high naturalness.

[0151] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0152] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0153] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the voice processing method described in any of the above embodiments.

[0154] Figure 6 This embodiment illustrates a more specific hardware structure of an electronic device. The device may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0155] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0156] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0157] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0158] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0159] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0160] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0161] The electronic devices described above are used to implement the corresponding voice processing methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0162] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to execute the speech processing method as described in any of the above embodiments.

[0163] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0164] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the speech processing method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0165] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a computer program product, which includes a computer program. In some embodiments, the computer program is executable by one or more processors to cause the processors to perform the speech processing method. Corresponding to the execution entity for each step in each embodiment of the method, the processor executing the corresponding step may belong to the corresponding execution entity.

[0166] The computer program product of the above embodiments is used to cause the processor to execute the speech processing method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0167] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, method, or computer program product. Therefore, this disclosure can be implemented as entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this disclosure can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0168] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (not exhaustive) of a computer-readable storage medium may include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0169] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0170] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0171] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0172] It should be understood that each block of a flowchart and / or block diagram, as well as combinations of blocks in a flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine that, when executed by a computer or other programmable data processing device, creates means for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0173] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce a product comprising an instruction apparatus that implements the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0174] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions that execute on the computer or other programmable apparatus can provide a process for implementing the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0175] Furthermore, although the operations of the methods of this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be executed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0176] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0177] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0178] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0179] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this disclosure, the provided drawings may or may not show well-known power / ground connections to integrated circuit (IC) chips and other components. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this disclosure, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this disclosure will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this disclosure can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0180] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0181] This disclosure is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A speech processing method, comprising: Get the first audio recording; Extract a first feature and at least one second feature from the first speech; Obtain a third feature that belongs to the same category as the first feature; A second speech is generated based on the third feature and at least one second feature.

2. The method according to claim 1, wherein, The first feature includes a first timbre feature for characterizing the timbre of the first speech, and the third feature includes a second timbre feature, which is different from the first timbre feature.

3. The method according to claim 2, wherein, The acquisition of a third feature of the same category as the first feature includes: From a preset set of timbre features, obtain timbre features whose similarity to the first timbre feature meets a preset threshold range, and determine the second timbre feature based on the timbre feature.

4. The method according to claim 2, wherein, The at least one second feature includes audio features and fundamental frequency features, and the generation of second speech based on the third feature and the at least one second feature includes: The audio features are fused with the second timbre features to obtain the first fused feature; The first fused feature and the fundamental frequency feature are fused to obtain the second fused feature; The second speech is generated based on the second fusion feature.

5. The method according to claim 4, wherein, The step of fusing the audio features with the second timbre features to obtain the first fused feature includes: Obtain the first statistical feature of the audio feature in the time dimension and the second statistical feature of the second timbre feature in the time dimension; The first statistical feature is normalized based on the second statistical feature to obtain the first fused feature.

6. The method according to claim 5, wherein, The step of fusing the first fused feature with the fundamental frequency feature to obtain the second fused feature includes: Obtain the third statistical feature of the first fusion feature in the time dimension and the fourth statistical feature of the fundamental frequency feature in the time dimension; The third statistical feature is normalized based on the fourth statistical feature to obtain the second fusion feature.

7. The method according to any one of claims 1-6, wherein, Input the first speech into the first model, and obtain the second speech output by the first model; The first model includes: a feature extraction module, a feature processing module, and a speech generation module; The feature extraction module is used to extract the first feature and the at least one second feature from the first speech; The feature processing module is used to output the third feature, which belongs to the same category as the first feature; The speech generation module is used to generate the second speech based on the third feature and at least one second feature.

8. The method according to claim 7, wherein, The speech generation module includes a fusion module and an audio reconstruction module: The fusion module is used to fuse the third feature and the at least one second feature to obtain a third fused feature; The audio reconstruction module is used to reconstruct the audio from the third fusion feature to generate the second speech.

9. The method according to claim 7, wherein, The training process of the first model includes: Obtain a test set for the first model, the test set including multiple test samples, each test sample including a third speech; The third speech is input into the first model to obtain the fourth speech; wherein, the first model includes the feature extraction module and the speech generation module, the feature extraction module is used to extract a fourth feature and at least one fifth feature from the third speech, and the speech generation module is used to generate the fourth speech based on the fourth feature and the at least one fifth feature; Obtain the loss information between the fourth speech and the third speech, perform backpropagation based on the loss information, and update the parameters of the first model.

10. The method according to claim 9, wherein, The fourth feature includes a third timbre feature used to characterize the timbre of the third speech.

11. The method according to claim 9, wherein, The speech generation module includes a fusion module and an audio reconstruction module; The fusion module is used to fuse the fourth feature and at least one fifth feature to obtain a fourth fused feature; The audio reconstruction module is used to reconstruct the audio from the fourth fusion feature and generate the fourth speech.

12. The method according to claim 11, wherein, The step of obtaining the first loss information between the fourth speech and the third speech includes: The loss information is calculated based on the loss function; The loss function includes the sum of reconstruction loss and generator loss. The reconstruction loss is obtained based on the average absolute error between the fourth speech and the third speech, and the generator loss is obtained based on the loss during the audio reconstruction process.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the voice processing method as described in any one of claims 1 to 12.

14. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the speech processing method according to any one of claims 1 to 12.

15. A computer program product, characterized in that, It includes computer program instructions that, when executed on a computer, cause the computer to perform the speech processing method as described in any one of claims 1 to 12.

Images