Voice conversion method and device, electronic equipment and storage medium

Abstract

The embodiment of the application provides a voice conversion method and device, electronic equipment and a storage medium, and belongs to the technical field of digital medical treatment. The method comprises the following steps: obtaining sample voice data and reference voice data; processing the reference voice data through a target timbre encoder to obtain a reference timbre vector; processing the sample voice data through an original content encoder to obtain a sample content vector; processing the reference timbre vector and the sample content vector through a target decoder to obtain sample reconstruction voice data; processing the sample reconstruction voice data through the original content encoder to obtain a reconstruction content vector; adjusting an original voice model according to the reconstruction content vector and the sample content vector to obtain a target voice model; and performing voice conversion on pre-acquired target voice data through the target voice model and preset target conversion object data to obtain converted voice data. The embodiment of the application can improve the accuracy of voice conversion.

Description

Technical Field

[0001] This application relates to the field of digital medical technology, and in particular to a speech conversion method and apparatus, electronic device and storage medium. Background Technology

[0002] Speech conversion refers to converting the speech of a source speaker into the speech of a target speaker without altering the content of the source speaker's speech. For example, in digital healthcare scenarios, during intelligent diagnosis, remote consultations, and online consultations, speech conversion can be performed to protect the privacy of the person being consulted, such as converting the speaker's speech into the speech of person A. In related technologies, the source speaker's (i.e., the person being consulted) speech includes speech content features and timbre features. Since there is a correlation between these features, directly converting speech based on the speech features of the target speaker (i.e., person A) often results in low accuracy. Therefore, improving the accuracy of speech conversion has become a pressing technical problem. Summary of the Invention

[0003] The main objective of this application is to provide a speech conversion method, apparatus, electronic device, and storage medium, which aims to improve the accuracy of speech conversion.

[0004] To achieve the above objectives, a first aspect of this application proposes a speech conversion method, the method comprising:

[0005] Acquire sample speech data and reference speech data;

[0006] The sample speech data and the reference speech data are input into a preset original speech model; wherein, the original speech model includes a target timbre encoder, an original content encoder, and a target decoder;

[0007] The reference speech data is processed by the target timbre encoder to obtain a reference timbre vector;

[0008] The original content encoder performs content encoding on the sample speech data to obtain a sample content vector;

[0009] The target decoder performs speech reconstruction processing on the reference timbre vector and the sample content vector to obtain sample reconstructed speech data.

[0010] The original content encoder is used to perform content encoding on the sample reconstructed speech data to obtain a reconstructed content vector.

[0011] The original speech model is adjusted according to the reconstructed content vector and the sample content vector to obtain the target speech model.

[0012] The target speech data is converted by using the target speech model and the preset target conversion object data to convert the pre-acquired target speech data.

[0013] In some embodiments, the target timbre encoder includes convolutional layers, attention layers, fully connected layers, and normalization layers;

[0014] The step of performing timbre encoding processing on the reference speech data through the target timbre encoder to obtain a reference timbre vector includes:

[0015] The convolutional layer is used to extract features from the reference speech data to obtain reference speech features;

[0016] Attention features are obtained by performing attention processing on the reference speech features through the attention layer;

[0017] The attention features are transformed by feature dimension conversion through the fully connected layer to obtain the transformed timbre features;

[0018] The normalization layer is used to normalize the converted timbre features to obtain the reference timbre vector.

[0019] In some embodiments, the attention layer includes an attention sublayer and a pooling sublayer;

[0020] The process of performing attention processing on the reference speech features through the attention layer to obtain attention features includes:

[0021] Attention weights are obtained by performing attention processing on the reference speech features through the attention sublayer.

[0022] The attention weights and the reference speech features are pooled by the pooling sub-layer to obtain average speech features and discrete speech features.

[0023] The attention features are obtained by performing feature fusion processing on the average speech features and the discrete speech features.

[0024] In some embodiments, adjusting the parameters of the original speech model based on the reconstructed content vector and the sample content vector to obtain the target speech model includes:

[0025] A similarity value is obtained by calculating the similarity between the reconstructed content vector and the sample content vector.

[0026] The parameters of the original content encoder are adjusted based on the similarity value to obtain the target content encoder;

[0027] The target speech model is obtained based on the target content encoder, the target timbre encoder, and the target decoder.

[0028] In some embodiments, the step of performing content encoding processing on the sample speech data using the original content encoder to obtain a sample content vector includes:

[0029] The sample speech data is subjected to speech perturbation processing to obtain perturbed speech data;

[0030] The perturbation speech data is processed by the original content encoder to obtain the sample content vector.

[0031] In some embodiments, the step of converting the pre-acquired target speech data using the target speech model and preset target conversion object data to obtain converted speech data includes:

[0032] The target speech data is reconstructed by using the target speech model and preset target conversion object data to obtain target speech reconstruction data.

[0033] The target speech reconstruction data is processed by a preset vocoder to obtain the converted speech data.

[0034] In some embodiments, before inputting the sample speech data and the reference speech data into a preset original speech model, the method further includes training the original speech model, specifically including:

[0035] Acquire training speech data;

[0036] The training speech data is input into a preset initial speech model; wherein the initial speech model includes an initial timbre encoder, an initial content encoder, and an initial decoder;

[0037] The initial timbre encoder is used to perform timbre encoding on the training speech data to obtain a training timbre vector;

[0038] The training speech data is processed by the initial content encoder to obtain a training content vector;

[0039] The training timbre vector and the training content vector are processed by the initial decoder to reconstruct the speech data.

[0040] The original speech model is obtained by reconstructing the speech data based on the training and adjusting the parameters of the initial speech model using the training speech data.

[0041] To achieve the above objectives, a second aspect of this application provides a speech conversion device, the device comprising:

[0042] The data acquisition module is used to acquire sample speech data and reference speech data;

[0043] A data input module is used to input the sample speech data and the reference speech data into a preset original speech model; wherein, the original speech model includes a target timbre encoder, an original content encoder, and a target decoder;

[0044] A timbre encoding module is used to perform timbre encoding processing on the reference speech data through the target timbre encoder to obtain a reference timbre vector;

[0045] The first content encoding module is used to perform content encoding processing on the sample speech data through the original content encoder to obtain a sample content vector.

[0046] The speech reconstruction module is used to perform speech reconstruction processing on the reference timbre vector and the sample content vector through the target decoder to obtain sample reconstructed speech data.

[0047] The second content encoding module is used to perform content encoding processing on the sample reconstructed speech data through the original content encoder to obtain a reconstructed content vector.

[0048] The parameter adjustment module is used to adjust the parameters of the original speech model according to the reconstructed content vector and the sample content vector to obtain the target speech model.

[0049] The speech conversion module is used to convert the pre-acquired target speech data into speech data using the target speech model and preset target conversion object data.

[0050] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0051] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0052] The speech conversion method, apparatus, electronic device, and storage medium proposed in this application perform content encoding processing on the acquired sample speech data using an original content encoder to obtain a sample content vector. Then, a target timbre encoder performs timbre encoding processing on the acquired reference speech data to obtain a reference timbre vector. Subsequently, a target decoder performs speech reconstruction processing on the reference timbre vector and the sample content vector, and the original content encoder processes the reconstructed sample speech data to obtain a reconstructed content vector. Since the reconstructed sample speech data is obtained from the output data of the target timbre encoder and the original content encoder, ideally, the reconstructed content vector obtained from the reconstructed sample speech data should have the same semantics as the output data of the original content encoder. Therefore, by adjusting the parameters of the original speech model using the reconstructed content vector and the sample content vector, the extraction error of speech content features such as the reconstructed content vector and the sample content vector by the original speech model can be continuously reduced, thereby achieving decoupling between speech content features and speech timbre features, and thus improving the accuracy of speech conversion. When this application is applied to scenarios such as intelligent diagnosis and treatment, remote consultation, and online consultation in digital healthcare, it can improve the accuracy of speech conversion of the person being questioned, and protect the privacy of the person being questioned while minimizing the impact on the content of the question. Attached Figure Description

[0053] Figure 1 This is a flowchart of the speech conversion method provided in the embodiments of this application;

[0054] Figure 2 yes Figure 1 The flowchart of the steps is also included before step S102;

[0055] Figure 3 This is a schematic diagram of the steps for adjusting the parameters of the original speech model;

[0056] Figure 4 yes Figure 1 The flowchart of step S103 in the process;

[0057] Figure 5 yes Figure 4 The flowchart of step S402 in the document;

[0058] Figure 6 yes Figure 1 The flowchart of step S106 in the process;

[0059] Figure 7 yes Figure 1 The flowchart of step S107 in the process;

[0060] Figure 8 yes Figure 1The flowchart of step S108 in the process;

[0061] Figure 9 This is a schematic diagram of the structure of the speech conversion device provided in the embodiments of this application;

[0062] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0066] First, let's analyze some of the terms used in this application:

[0067] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0068] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0069] Pre-emphasis is a preprocessing technique used in feature extraction. Since the energy of high-frequency signals in speech signals is typically low and subject to significant suppression, it is necessary to increase the energy of the high-frequency components, i.e., compensate for the amplitude of the high-frequency components of the speech signal. This makes the energy distribution of the speech signal more balanced, thereby preventing instability in the numerical calculation of the Fourier transform.

[0070] Framing is a preprocessing method in feature extraction. Speech signals are non-stationary, but considering the regular rhythmic movement of the vocal cords during voiced speech, meaning the fundamental frequency is relatively fixed within a short time range, speech signals exhibit short-time stationarity. Specifically, the speech signal within 10ms to 30ms is considered a quasi-stationary process. Short-time analysis primarily uses framing, but the fundamental frequency may change between adjacent frames. To ensure the smoothness of acoustic feature parameters, overlapping frames are generally used. Specifically, the speech signal is segmented while maintaining a certain repetition rate. Framing is equivalent to adding a rectangular window to the speech signal; therefore, the main parameters of framing include frame length (the length of each frame) and frame shift (the distance the rectangular window moves).

[0071] Windowing is a preprocessing technique used in feature extraction. Considering the short-term stationarity of speech signals, windowing is applied to each frame of the signal. Common window functions include the Hamming window, Hanning window, and Blackman window. Specifically, after dividing the speech signal into frames, each frame is substituted into the window function, with the values ​​outside the window set to 0 to eliminate potential signal discontinuities at the ends of each frame. For example, a Hamming window of size n is applied to each m frames of the signal. By applying a Hamming window to each frame of data, the Hamming window matrix C(m, n) is obtained.

[0072] Fourier Transform: This refers to the ability to represent a function satisfying certain conditions as a trigonometric function (sine or cosine function) or a linear combination of their integrals. In feature extraction, a Fast Fourier Transform (FFT) of N = 4*n is performed on each frame, resulting in a matrix of size (m, 4n). The number of frames after the Fourier Transform is still m. The modulus of the distribution of the 4n data points in each frame is taken, squared, and then divided by 4n to obtain the energy spectral density E(m, 4n), which has a size of m*4n. Then, the energy obtained from each frame is summed to obtain an m*1 matrix, where each element represents the sum of the energy in one frame.

[0073] Mel filter bank: Defines a filter bank with M filters, where the number of filters is similar to the number of critical bands, and M is typically 22-26. The energy spectrum of a speech signal is processed using a set of triangular filters with a Mel scale, constructed as described above, to extract features from the speech signal. The triangular filters have the following functions: First, they are dense at low frequencies and sparse at high frequencies, thus mimicking the high resolution of low-frequency signals by the human ear. Second, they smooth the spectrum and eliminate harmonics, thereby highlighting the formants of the original speech signal. It can be understood that the spectrum has an envelope and fine structure, corresponding to timbre and pitch, respectively. For speech signals, timbre is the primary useful information; integration within the triangle can eliminate the fine structure, retaining only timbre information. Third, the sequence obtained from the Fourier transform is relatively long; the triangular filters transform it into the energy under each triangle, thereby reducing the amount of data.

[0074] Speech conversion refers to converting the speech of a source speaker into the speech of a target speaker without altering the content of the source speaker's speech. For example, in digital healthcare scenarios, during intelligent diagnosis, remote consultations, and online consultations, speech conversion can be performed to protect the privacy of the person being consulted, such as converting the speaker's speech into the speech of person A. In related technologies, the source speaker's (i.e., the person being consulted) speech includes speech content features and timbre features. Since there is a correlation between these features, directly converting speech based on the speech features of the target speaker (i.e., person A) often results in low accuracy. Therefore, improving the accuracy of speech conversion has become a pressing technical problem.

[0075] Based on this, embodiments of this application provide a speech conversion method and apparatus, an electronic device and a storage medium, aiming to improve the accuracy of speech conversion.

[0076] The speech conversion method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the speech conversion method in this application is described.

[0077] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0078] Foundational technologies in artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0079] The speech conversion method provided in this application relates to the field of digital medical technology. The speech conversion method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speech conversion method, but is not limited to the above forms.

[0080] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0081] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user voice data, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0082] Figure 1 This is an optional flowchart of the speech conversion method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S108.

[0083] Step S101: Obtain sample speech data and reference speech data;

[0084] Step S102: Input the sample speech data and reference speech data into the preset original speech model; wherein, the original speech model includes a target timbre encoder, an original content encoder and a target decoder;

[0085] Step S103: Perform timbre encoding on the reference speech data using the target timbre encoder to obtain the reference timbre vector;

[0086] Step S104: Perform content encoding processing on the sample speech data using the original content encoder to obtain the sample content vector;

[0087] Step S105: The reference timbre vector and sample content vector are processed by the target decoder to obtain the sample reconstructed speech data;

[0088] Step S106: Perform content encoding processing on the sample reconstructed speech data using the original content encoder to obtain the reconstructed content vector;

[0089] Step S107: Adjust the parameters of the original speech model based on the reconstructed content vector and the sample content vector to obtain the target speech model;

[0090] Step S108: Use the target speech model and preset target conversion object data to convert the pre-acquired target speech data to obtain converted speech data.

[0091] Steps S101 to S108, as illustrated in this embodiment, involve content encoding of the acquired sample speech data using the original content encoder to obtain a sample content vector. Then, timbre encoding of the acquired reference speech data is performed using the target timbre encoder to obtain a reference timbre vector. Subsequently, speech reconstruction processing is performed on the reference timbre vector and the sample content vector using the target decoder, and the reconstructed speech data obtained from the speech reconstruction is then processed by the original content encoder to obtain a reconstructed content vector. Since the reconstructed speech data is obtained from the output data of the target timbre encoder and the original content encoder, ideally, the reconstructed content vector obtained from the reconstructed speech data should have the same semantic meaning as the output data of the original content encoder. Therefore, by adjusting the parameters of the original speech model using the reconstructed content vector and the sample content vector, the extraction error of the original speech model for speech content features such as the reconstructed content vector and the sample content vector can be continuously reduced, thereby achieving decoupling between speech content features and speech timbre features, and thus improving the accuracy of speech conversion. When this application is applied to scenarios such as intelligent diagnosis and treatment, remote consultation, and online consultation in digital healthcare, it can improve the accuracy of speech conversion of the person being questioned, and protect the privacy of the person being questioned while minimizing the impact on the content of the question.

[0092] In step S101 of some embodiments, sample speech data of a sample object is obtained through an Application Programming Interface (API) or similar means, and reference speech data of a reference object is also obtained. The sample object and the reference object are two different objects; that is, the speech representation features of the sample speech data differ from those of the reference speech data. Speech representation features include timbre features, content features, prosodic features, etc. For example, consider sample speech data. When this application is applied to a terminal, the terminal can have applications with voice input functions installed, such as instant messaging applications, voice input method applications, voice assistants, etc. Sample speech data of the sample object is obtained according to the API interface of the aforementioned applications. Furthermore, audio extraction can be performed on video data and audio data obtained through big data to obtain corresponding sample speech data. When this application is applied to a digital healthcare scenario, the sample speech data can be the speech data of an inquirer, such as speech data describing a symptom. Therefore, this application embodiment does not specifically limit the method of obtaining sample speech data.

[0093] Reference Figure 2 In some embodiments, before step S102, the speech conversion method provided in this application further includes training an original speech model, specifically including but not limited to steps S201 to S206.

[0094] Step S201: Obtain training speech data;

[0095] Step S202: Input the training speech data into the preset initial speech model; wherein, the initial speech model includes an initial timbre encoder, an initial content encoder, and an initial decoder;

[0096] Step S203: Perform timbre encoding on the training speech data using the initial timbre encoder to obtain the training timbre vector;

[0097] Step S204: Perform content encoding processing on the training speech data using the initial content encoder to obtain the training content vector;

[0098] Step S205: Perform speech reconstruction processing on the training timbre vector and training content vector through the initial decoder to obtain training reconstructed speech data;

[0099] Step S206: Reconstruct the speech data based on the training data and adjust the parameters of the initial speech model to obtain the original speech model.

[0100] In step S201 of some embodiments, training speech data of the training object is obtained through a relevant API interface. It is understood that the method of obtaining the training speech data is the same as the method of obtaining the sample speech data; that is, the training speech data is the speech data of the training object, such as speech data describing a symptom, in which case the training object is the object being questioned. This will not be elaborated further in the embodiments of this application.

[0101] In step S202 of some embodiments, an initial speech model based on an encoder-decoder structure is pre-set, that is, the initial speech model includes an initial timbre encoder, an initial content encoder, and an initial decoder. Training speech data is used as input data for the initial speech model to extract speech representation features from the training speech data through the initial speech model.

[0102] In step S203 of some embodiments, the training speech data is used as input data for the target timbre encoder, so that the timbre features in the training speech data are extracted and encoded by the target timbre encoder to obtain a training timbre vector that represents the identity features of the training object.

[0103] In step S204 of some embodiments, the training speech data is used as input data for the initial content encoder, so that the content information features to be expressed by the training speech data are extracted and encoded by the initial content encoder to obtain the training content vector.

[0104] In step S205 of some embodiments, the training timbre vector and the training content vector are used as input data to the initial decoder, so that the initial decoder performs speech reconstruction processing on the training timbre vector and the training content vector to obtain training reconstructed speech data. That is, the training reconstructed speech data is data whose timbre features are the training timbre vector and whose content features are the training content vector.

[0105] In step S206 of some embodiments, since the training reconstructed speech data is data reconstructed from the speech representation features of the training speech data, ideally, the training reconstructed speech data should be identical to the training speech data. Therefore, a loss calculation can be performed on the training reconstructed speech data and the training speech data according to a preset loss function, so as to determine the error of the initial speech model in extracting speech representation features based on the calculated loss value, thereby training an original speech model with more accurate speech representation feature extraction capabilities. It is understood that different parameter adjustment ratios can be set according to actual needs. For example, in the embodiments of this application, the initial timbre encoder and the initial decoder are mainly adjusted. This embodiment of the application does not specifically limit this aspect.

[0106] In step S102 of some embodiments, a raw speech model including a target timbre encoder, a raw content encoder, and a target decoder is pre-constructed. Sample speech data and reference speech data are respectively input into the raw speech model to extract speech representation features from the sample speech data and the reference speech data. It is understood that in the raw speech model, the target timbre encoder and the target decoder are pre-trained models; that is, the target timbre encoder already has a certain degree of accuracy in timbre encoding, and the target decoder already has a certain degree of accuracy in speech reconstruction processing.

[0107] In step S103 of some embodiments, referring to Figure 3 The reference speech data is used as input data to the target timbre encoder. The target timbre encoder extracts and encodes the timbre features from the reference speech data to obtain a reference timbre vector. For example, when the goal is to convert all speech into speech with the timbre of object A, the reference timbre vector is the timbre vector of object A. It is understood that timbre features are features that reflect the speech characteristics of the reference object (i.e., object A). Timbre features are related to the identity of the reference object; that is, speech data corresponding to reference objects and sample objects with different identities have different timbre features. It is understood that object A can be an elderly person, a child, an anime character, etc., and this embodiment of the application does not specifically limit this.

[0108] Reference Figure 4 In some embodiments, the target tone encoder includes a convolutional layer, an attention layer, a fully connected layer, and a normalization layer. Step S103 includes, but is not limited to, steps S401 to S404.

[0109] Step S401: Extract features from the reference speech data using a convolutional layer to obtain the reference speech features;

[0110] Step S402: Perform attention processing on the reference speech features through an attention layer to obtain attention features;

[0111] Step S403: Perform feature dimension transformation on the attention features through a fully connected layer to obtain the transformed timbre features;

[0112] Step S404: Normalize the converted timbre features through a normalization layer to obtain a reference timbre vector.

[0113] In step S401 of some embodiments, a convolutional layer including multiple convolutional kernels and offset values ​​is pre-set, wherein the convolutional kernels are used to extract local features from the reference speech data input to the convolutional layer. The extracted local features and offset values ​​are calculated to obtain the reference speech features. It can be understood that, in order to reduce the amount of data input to the target tone encoder, the reference speech data can also be subjected to dimensionality reduction processing. Specifically, when the reference speech data is time-domain data, the reference speech data is processed by pre-emphasis, framing, windowing, Fourier transform, Mel filter bank, etc., to obtain the frequency domain representation of the reference speech data, that is, the Mel spectrum.

[0114] In step S402 of some embodiments, the reference speech features output by the convolutional layer are used as input data for the attention layer, so that the reference speech features are processed by the attention layer. It is understood that in a segment of speech data, certain frame features are usually more important than others, meaning they better represent the speech characteristics of that segment of speech data. Therefore, by assigning different attention weights to these frame features through the attention layer, attention features that consider both the correlation between global features and the importance of local features are obtained.

[0115] Reference Figure 5 In some embodiments, the attention layer includes an attention sublayer and a pooling sublayer. Step S402 includes, but is not limited to, steps S501 to S503.

[0116] Step S501: Perform attention processing on the reference speech features through an attention sublayer to obtain attention weights;

[0117] Step S502: The attention weights and reference speech features are pooled through a pooling sub-layer to obtain the average speech features and discrete speech features.

[0118] Step S503: Perform feature fusion processing on the average speech features and discrete speech features to obtain attention features.

[0119] In step S501 of some embodiments, in a segment of speech data, certain frame features are usually more important than other frame features, that is, they are more representative of the speech characteristics of the segment of speech data. Therefore, different attention weights are assigned to each frame feature represented by the reference speech features through an attention sublayer.

[0120] In step S502 of some embodiments, the attention weights output by the attention sublayer and the reference speech features are used as input data for the pooling sublayer. Each reference speech feature is weighted according to the attention weights, and then the average value of the reference speech features is calculated to determine the average characteristics of all reference speech features, thus obtaining the average speech features. Furthermore, the labeled difference is calculated on the reference speech features to determine the degree of dispersion of all reference speech features, thus obtaining the discrete speech features.

[0121] In step S503 of some embodiments, the average speech features and discrete speech features are fused, and the attention features obtained after fusion are used as input data for the fully connected layer.

[0122] In step S403 of some embodiments, the attention features output by the attention layer are used as input data for the fully connected layer, so as to integrate multiple attention features through the fully connected layer and convert the integrated features into one-dimensional features to obtain the converted timbre features.

[0123] In step S404 of some embodiments, the converted timbre features output by the fully connected layer are used as input data for the normalization layer to map the converted timbre features to a preset interval to obtain a reference timbre vector. This can eliminate the influence of abnormal features on the target timbre encoder and improve the target timbre model's ability to extract timbre features from the reference speech data.

[0124] In step S104 of some embodiments, the sample speech data is used as input data to the original content encoder, so that the original content encoder can extract and encode the content information features to be expressed by the sample speech data to obtain a sample content vector. For example, if the content information to be expressed by the sample speech data is "I woke up this morning with a headache and weakness in my limbs", then the sample content vector is a feature vector corresponding to the above content information.

[0125] Reference Figure 6 In some embodiments, step S106 includes, but is not limited to, steps S601 to S602.

[0126] Step S601: Perform speech perturbation processing on the sample speech data to obtain perturbed speech data;

[0127] Step S602: Perform content encoding processing on the perturbed speech data using the original content encoder to obtain the sample content vector.

[0128] In step S601 of some embodiments, in order to improve the capability of the original content encoder, even if the original content encoder extracts an accurate sample content vector from the sample speech data, speech perturbation processing is performed on the sample speech data to obtain perturbed speech data. Specifically, the speech perturbation processing operation refers to the operation used to change the prosodic features in the sample speech data. Among them, prosodic features include pitch features, rhythm features, etc.

[0129] In step S602 of some embodiments, the perturbed speech data is used as input data to the original content encoder, so that the original content encoder can extract content features from the perturbed speech data to obtain a sample content vector. During the training of the original speech model, the original content encoder continuously updates its ability to mine real content features from the perturbed speech data, that is, updates its ability to avoid interference from perturbed features, thereby improving the accuracy of the original content encoder in extracting content features.

[0130] In step S105 of some embodiments, referring to Figure 2 The reference timbre vector and sample content vector are used as input data to the target decoder, so that the target decoder can perform speech reconstruction processing on the reference timbre vector and sample content vector to obtain sample reconstructed speech data.

[0131] In step S106 of some embodiments, the sample reconstructed speech data is used as input data of the original content encoder so that the original content encoder can extract and encode the content information features to be expressed by the sample reconstructed speech data to obtain the reconstructed content vector.

[0132] In step S107 of some embodiments, ideally, the sample reconstructed speech data obtained by the target decoder should have the same timbre features as the reference speech data and the same content information features as the sample speech data. That is, ideally, when the sample reconstructed speech data is input to the original content encoder, the reconstructed content vector output by the original content encoder should have the same semantics as the sample content vector. Taking the example from the above embodiment, ideally, the speech content of the sample reconstructed speech should still be "I woke up this morning with a headache and weakness in my limbs," but the timbre of the sample reconstructed speech should be the timbre of object A. Therefore, a preset loss function can be used to calculate the loss between the reconstructed content vector and the sample content vector, and the parameters of the original speech model can be adjusted based on the calculated loss value to obtain the target speech model.

[0133] Reference Figure 7 In some embodiments, step S107 includes, but is not limited to, steps S701 to S703.

[0134] Step S701: Calculate the similarity based on the reconstructed content vector and the sample content vector to obtain the similarity value;

[0135] Step S702: Adjust the parameters of the original content encoder based on the similarity value to obtain the target content encoder;

[0136] Step S703: Obtain the target speech model based on the target content encoder, target timbre encoder, and target decoder.

[0137] It should be noted that, since the loss calculation is performed on the reconstructed content vector and the sample content vector, and the target timbre encoder and target decoder are pre-trained models, the calculated loss value in this embodiment is mainly used to adjust the parameters of the original content encoder.

[0138] In step S701 of some embodiments, a similarity calculation is performed on the reconstructed content vector and the sample content vector to determine the distance between the reconstructed content vector and the sample content vector in the vector space, thereby obtaining a similarity value. Specifically, any of the following methods can be selected for similarity calculation: Euclidean distance, Manhattan distance, Hamming distance, cosine similarity, Chebyshev distance, etc., and this application embodiment does not specifically limit the method.

[0139] In step S702 of some embodiments, the parameters of the original content encoder are adjusted according to the calculated similarity value and the backpropagation algorithm so that the preset loss function converges, thereby obtaining a target content encoder with more accurate content information feature extraction.

[0140] In step S703 of some embodiments, a target speech model is constructed based on the target content encoder, target timbre encoder, and target decoder obtained after parameter adjustment.

[0141] In step S108 of some embodiments, target speech data of the target conversion object is acquired, features are extracted from the target speech data according to the target speech model, and the extracted features are replaced with target conversion object data. The target conversion object data refers to the speech feature data of the target conversion object, such as timbre feature data. Therefore, according to the above operations, target speech data can be converted into speech data with the speech feature data of the target conversion object, i.e., speech data is converted, thereby achieving speech conversion. Furthermore, target speech data of the target conversion object can be acquired through API interfaces or other means. For example, when this application is applied to a digital healthcare scenario, target speech data can be acquired through intelligent diagnosis API interfaces, remote consultation API interfaces, online consultation API interfaces, etc. When this application is applied to a terminal, the terminal can install applications with voice input functions, such as instant messaging applications, voice input method applications, voice assistants, etc. The target conversion object can use these applications to input the target speech data to be converted. It should be noted that in practical applications, when the target conversion object uses an instant messaging application for speech conversion, it can input the target speech data by clicking the voice conversion interface of the instant messaging application and using the voice input control set on that interface. At this point, the instant messaging application obtains the target voice data by calling a voice acquisition device, such as turning on the microphone.

[0142] It is understandable that the target conversion object data can also be identity identification data. A voice database is pre-set, which includes voice feature data that has a mapping relationship with the identity identification data. When the target conversion object uses an instant messaging application for voice conversion, or performs voice conversion using a preset voice conversion interface in a digital medical scenario, the voice conversion interface can also be set with identity identification controls corresponding to different identity identification data. By obtaining the instruction triggered by the target conversion object to generate the identity identification control, the corresponding voice feature data is obtained from the voice database according to the instruction, thereby realizing the voice conversion of the target voice data based on the voice feature data. For example, controls for "female voice", "male voice", "child's voice", "elderly voice", etc. are set. If the target conversion object is female, when the instruction triggered by the target conversion object to use the "male voice" control is obtained, the voice timbre of the target voice data can be converted from the female voice timbre corresponding to the target conversion object to the male voice timbre without changing the content of the symptoms described by the target conversion object. Therefore, it can be seen that the embodiments of this application do not specifically limit the data type and acquisition method of the target conversion object data. In addition to privacy protection, the converted voice data can also be used to assist applications such as car radio and announcements, car navigation, electronic dictionaries, consumer electronics, smartphones, smart speakers, voice assistants, and e-book reading.

[0143] Reference Figure 8 In some embodiments, step S108 includes, but is not limited to, steps S801 to S802.

[0144] Step S801: Perform speech reconstruction processing on the target speech data using the target speech model and preset target conversion object data to obtain target speech reconstruction data;

[0145] Step S802: The target speech reconstruction data is processed by a preset vocoder to obtain converted speech data.

[0146] In step S801 of some embodiments, the target speech model includes a target content encoder, a target timbre encoder, and a target decoder. The target speech data is used as input data to the target content encoder to obtain speech content features of the target speech data. The target conversion object data and the obtained speech content features are used as input data to the target decoder to reconstruct new speech data, i.e., target speech reconstruction data, through the target decoder. It is understood that the method of obtaining the target conversion object data and the data type of the target conversion object data can be referred to the description in the above embodiments, and will not be repeated in this application.

[0147] In some embodiments, in step S802, since the data output by the target decoder is in the form of Mel spectrum, the target speech reconstruction data also needs to be input to a preset vocoder to output audio data corresponding to the target speech reconstruction data, thereby realizing the speech conversion of the target speech data.

[0148] The speech conversion method provided in this application involves encoding the acquired sample speech data using an original content encoder to obtain a sample content vector. Then, a target timbre encoder encodes the acquired reference speech data to obtain a reference timbre vector. Next, a target decoder performs speech reconstruction processing on the reference timbre vector and the sample content vector. Finally, the original content encoder processes the reconstructed speech data using the original content encoder to obtain a reconstructed content vector. Since the reconstructed speech data is derived from the output data of the target timbre encoder and the original content encoder, ideally, the reconstructed content vector should have the same semantic meaning as the output data of the original content encoder. Therefore, by adjusting the parameters of the original content encoder using the reconstructed content vector and the sample content vector, the extraction error of speech content features such as the reconstructed content vector and the sample content vector by the original content encoder can be continuously reduced. This decoupling between speech content features and speech timbre features improves the accuracy of speech conversion when using the target speech model constructed from the target content encoder, target timbre encoder, and target decoder. When this application is applied to scenarios such as intelligent diagnosis and treatment, remote consultation, and online consultation in digital healthcare, it can improve the accuracy of speech conversion of the person being questioned, and protect the privacy of the person being questioned while minimizing the impact on the content of the question.

[0149] Please see Figure 9 This application also provides a speech conversion device that can implement the above-described speech conversion method. The device includes:

[0150] Data acquisition module 901 is used to acquire sample speech data and reference speech data;

[0151] The data input module 902 is used to input sample speech data and reference speech data into a preset original speech model; wherein, the original speech model includes a target timbre encoder, an original content encoder and a target decoder;

[0152] The timbre encoding module 903 is used to perform timbre encoding processing on the reference speech data through the target timbre encoder to obtain the reference timbre vector;

[0153] The first content encoding module 904 is used to perform content encoding processing on the sample speech data through the original content encoder to obtain the sample content vector.

[0154] The speech reconstruction module 905 is used to perform speech reconstruction processing on the reference timbre vector and the sample content vector through the target decoder to obtain sample reconstructed speech data.

[0155] The second content encoding module 906 is used to perform content encoding processing on the sample reconstructed speech data through the original content encoder to obtain the reconstructed content vector.

[0156] The parameter adjustment module 907 is used to adjust the parameters of the original speech model based on the reconstructed content vector and the sample content vector to obtain the target speech model.

[0157] The speech conversion module 908 is used to convert the pre-acquired target speech data into speech data by using the target speech model and preset target conversion object data.

[0158] The specific implementation of this voice conversion device is basically the same as the specific implementation of the above-described voice conversion method, and will not be repeated here.

[0159] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described speech conversion method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0160] Reference Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0161] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0162] The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 using the speech conversion method of the embodiments of this application.

[0163] Input / output interface 1003 is used to implement information input and output;

[0164] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0165] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0166] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0167] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described speech conversion method.

[0168] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0169] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0170] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0171] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0173] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0174] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0176] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0178] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0179] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech conversion method, characterized in that, The method includes: Acquire sample speech data and reference speech data; The sample speech data and the reference speech data are input into a preset original speech model; wherein, the original speech model includes a target timbre encoder, an original content encoder, and a target decoder; The reference speech data is processed by the target timbre encoder to obtain a reference timbre vector; The original content encoder performs content encoding on the sample speech data to obtain a sample content vector; The target decoder performs speech reconstruction processing on the reference timbre vector and the sample content vector to obtain sample reconstructed speech data. The original content encoder is used to perform content encoding on the sample reconstructed speech data to obtain a reconstructed content vector. The original speech model is adjusted according to the reconstructed content vector and the sample content vector to obtain the target speech model. The target speech data is converted by using the target speech model and the preset target conversion object data to convert the pre-acquired target speech data.

2. The method according to claim 1, characterized in that, The target timbre encoder includes a convolutional layer, an attention layer, a fully connected layer, and a normalization layer; The step of performing timbre encoding processing on the reference speech data through the target timbre encoder to obtain a reference timbre vector includes: The convolutional layer is used to extract features from the reference speech data to obtain reference speech features; Attention features are obtained by performing attention processing on the reference speech features through the attention layer; The attention features are transformed by feature dimension conversion through the fully connected layer to obtain the transformed timbre features; The normalization layer is used to normalize the converted timbre features to obtain the reference timbre vector.

3. The method according to claim 2, characterized in that, The attention layer includes an attention sub-layer and a pooling sub-layer; The process of performing attention processing on the reference speech features through the attention layer to obtain attention features includes: Attention weights are obtained by performing attention processing on the reference speech features through the attention sublayer. The attention weights and the reference speech features are pooled by the pooling sub-layer to obtain average speech features and discrete speech features. The attention features are obtained by performing feature fusion processing on the average speech features and the discrete speech features.

4. The method according to claim 1, characterized in that, The step of adjusting the parameters of the original speech model based on the reconstructed content vector and the sample content vector to obtain the target speech model includes: A similarity value is obtained by calculating the similarity between the reconstructed content vector and the sample content vector. The parameters of the original content encoder are adjusted based on the similarity value to obtain the target content encoder; The target speech model is obtained based on the target content encoder, the target timbre encoder, and the target decoder.

5. The method according to any one of claims 1 to 4, characterized in that, The step of performing content encoding processing on the sample speech data through the original content encoder to obtain a sample content vector includes: The sample speech data is subjected to speech perturbation processing to obtain perturbed speech data; The perturbation speech data is processed by the original content encoder to obtain the sample content vector.

6. The method according to any one of claims 1 to 4, characterized in that, The step of converting the pre-acquired target speech data using the target speech model and preset target conversion object data to obtain converted speech data includes: The target speech data is reconstructed by using the target speech model and preset target conversion object data to obtain target speech reconstruction data. The target speech reconstruction data is processed by a preset vocoder to obtain the converted speech data.

7. The method according to any one of claims 1 to 4, characterized in that, Before inputting the sample speech data and the reference speech data into a preset original speech model, the method further includes training the original speech model, specifically including: Acquire training speech data; The training speech data is input into a preset initial speech model; wherein the initial speech model includes an initial timbre encoder, an initial content encoder, and an initial decoder; The initial timbre encoder is used to perform timbre encoding on the training speech data to obtain a training timbre vector; The training speech data is processed by the initial content encoder to obtain a training content vector; The training timbre vector and the training content vector are processed by the initial decoder to reconstruct the speech data. The original speech model is obtained by reconstructing the speech data based on the training and adjusting the parameters of the initial speech model using the training speech data.

8. A voice conversion device, characterized in that, The device includes: The data acquisition module is used to acquire sample speech data and reference speech data; A data input module is used to input the sample speech data and the reference speech data into a preset original speech model; wherein, the original speech model includes a target timbre encoder, an original content encoder, and a target decoder; A timbre encoding module is used to perform timbre encoding processing on the reference speech data through the target timbre encoder to obtain a reference timbre vector; The first content encoding module is used to perform content encoding processing on the sample speech data through the original content encoder to obtain a sample content vector. The speech reconstruction module is used to perform speech reconstruction processing on the reference timbre vector and the sample content vector through the target decoder to obtain sample reconstructed speech data. The second content encoding module is used to perform content encoding processing on the sample reconstructed speech data through the original content encoder to obtain a reconstructed content vector. The parameter adjustment module is used to adjust the parameters of the original speech model according to the reconstructed content vector and the sample content vector to obtain the target speech model. The speech conversion module is used to convert the pre-acquired target speech data into speech data using the target speech model and preset target conversion object data.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images