Voice data processing method and device

Abstract

The present invention discloses a method and device for processing speech data. The method comprises: obtaining a phoneme sequence of a sample speech of a target object; encoding the phoneme sequence according to a preset rule to obtain a phoneme coding sequence; recognizing the acoustic features of the sample speech through a preset speech conversion model to obtain the acoustic features of the target object, wherein the input of the speech conversion model is the phoneme coding sequence, and the output of the speech conversion model is the acoustic features of the sample speech; after receiving a speech conversion instruction, performing speech conversion on a sentence to be converted according to the acoustic features of the target object to obtain a speech that meets the features of the target object. Through the present invention, the effect of improving the naturalness of speech conversion is achieved.

Description

Technical Field

[0001] The present invention relates to the field of speech, and in particular to a speech data processing method and device. Background Art

[0002] Speech conversion refers to transferring the timbre of a sentence from the original speaker to the target speaker while keeping the language content unchanged. It can play an important role in movie dubbing, short video voice changing, virtual humans, etc.

[0003] Most current speech conversion technologies are based on parallel data (two speaker corpora with the same sentence content). However, in engineering practice, the cost of obtaining parallel data is very high, and it is not possible to complete the conversion of any speaker to a specified speaker. There are also some speech conversion methods based on non-parallel data that are being studied, but these methods are all based on the mapping of original and converted sentences with equal length. Since the speech speed, rhythm and other characteristics of different speakers vary greatly, the absolute equal length mapping will limit the naturalness of the converted speech and the improvement of the similarity with the target speaker.

[0004] Since the solution in the related technology requires two speaker corpora and the recorded content must be consistent, this greatly increases the data threshold for the experiment; the model models the mapping of acoustic features between the two corpora, so it can complete the conversion of acoustic features from the original speaker in the corpus to the target speaker. In fact, the acoustic feature sequence is closely related to the speaker's speech speed, rhythm and other information. It is difficult to model the acoustic feature mapping relationship from all speakers to the target speaker only through the limited information in the parallel data corpus, resulting in limited generalization ability of the model and inability to solve the conversion of any speaker to a specific target speaker. This method has limitations.

[0005] Speech recognition extracts acoustic posterior probability information from the target speaker's sentences frame by frame, so the speech conversion system completes the mapping between acoustic posterior probabilities of equal length and acoustic features. Because the acoustic posterior probability sequence extracted frame by frame contains the speaker's pronunciation duration information (the number of frames corresponding to each phoneme), the pronunciation features of the speech conversion system are completely dependent on the acoustic posterior probability sequence. In the speech conversion stage, if the prosodic information of the original speaker and the target speaker is quite different, the similarity of the converted speech will be greatly reduced, and the naturalness will also be impaired.

[0006] With regard to the problem of low naturalness of speech conversion in related technologies, no effective solution has been proposed so far. Summary of the invention

[0007] The main purpose of the present invention is to provide a method and device for processing speech data to solve the problem of low naturalness of speech conversion.

[0008] In order to achieve the above-mentioned purpose, according to one aspect of the present invention, a speech data processing method is provided, the method comprising: obtaining a phoneme sequence of a sample speech of a target object; encoding the phoneme sequence according to a preset rule to obtain a phoneme coding sequence; identifying the acoustic features of the sample speech through a preset speech conversion model to obtain the acoustic features of the target object, wherein the input of the speech conversion model is the phoneme coding sequence, and the output of the speech conversion model is the acoustic features of the sample speech; after receiving a speech conversion instruction, performing speech conversion on the sentence to be converted according to the acoustic features of the target object to obtain a speech that meets the features of the target object.

[0009] Furthermore, the acoustic features of the sample speech are identified through a preset speech conversion model to obtain the acoustic features of the target object, including: vector encoding the phoneme coding sequence of the sample speech through the encoder layer of the preset speech conversion model to obtain a coding vector; calculating the alignment similarity between the coding vector and the vector corresponding to the acoustic feature through the decoder layer; when the alignment similarity meets a preset condition, obtaining a mapping between the coding vector and the vector corresponding to the acoustic feature; and determining the mapping rule between the coding vector and the vector corresponding to the acoustic feature as the acoustic feature of the target object.

[0010] Furthermore, the length of the encoding vector is equal to the length of the phoneme encoding sequence, and the length of the output vector of the decoder layer is equal to the length of the sample speech.

[0011] Furthermore, after receiving the speech conversion instruction, the sentence to be converted is speech-converted according to the acoustic characteristics of the target object to obtain speech that meets the characteristics of the target object, including: extracting a phoneme sequence from the sentence to be converted and then encoding it to obtain a phoneme coding sequence of the sentence to be converted; inputting the phoneme coding sequence of the sentence to be converted into a trained speech conversion model to obtain characteristics that meet the target object, wherein the trained speech conversion model is obtained based on the preset speech conversion model, the input of the trained speech conversion model is a phoneme coding sequence, and the output of the trained speech conversion model is the language habit characteristics of the target object; and performing waveform restoration on the characteristics that meet the target object through an LPCNet vocoder to obtain speech that meets the characteristics of the target object.

[0012] Furthermore, after obtaining the voice that meets the characteristics of the target object, the method further includes: playing the voice that meets the characteristics of the target object on a preset device.

[0013] Furthermore, the acoustic features of the target object are 20-dimensional acoustic features, including 18-dimensional Bark cepstral coefficients, 1-dimensional fundamental frequency inverse, and 1-dimensional correlation coefficient.

[0014] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, a speech data processing device is further provided, which includes: an acquisition unit, used to acquire a phoneme sequence of a sample speech of a target object; an encoding unit, used to encode the phoneme sequence according to a preset rule to obtain a phoneme encoding sequence; a recognition unit, used to recognize the acoustic features of the sample speech through a preset speech conversion model to obtain the acoustic features of the target object, wherein the input of the speech conversion model is the phoneme encoding sequence, and the output of the speech conversion model is the acoustic features of the sample speech; a conversion unit, used to perform speech conversion on the sentence to be converted according to the acoustic features of the target object after receiving a speech conversion instruction, to obtain speech that meets the features of the target object.

[0015] Furthermore, the recognition unit includes: an encoding module, which is used to vector encode the phoneme encoding sequence of the sample speech through the encoder layer of a preset speech conversion model to obtain a coding vector; a calculation module, which is used to calculate the alignment similarity between the coding vector and the vector corresponding to the acoustic feature through a decoder layer; a mapping module, which is used to obtain the mapping between the coding vector and the vector corresponding to the acoustic feature when the alignment similarity meets a preset condition; and a determination module, which is used to determine the mapping rule between the coding vector and the vector corresponding to the acoustic feature as the acoustic feature of the target object.

[0016] In order to achieve the above objective, according to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed by a processor, the speech data processing method of the present invention is executed.

[0017] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, an electronic device is also provided, characterized in that it includes: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the voice data processing method described in the present invention.

[0018] The present invention obtains the phoneme sequence of the sample speech of the target object; encodes the phoneme sequence according to a preset rule to obtain a phoneme coding sequence; recognizes the acoustic features of the sample speech through a preset speech conversion model to obtain the acoustic features of the target object, wherein the input of the speech conversion model is the phoneme coding sequence, and the output of the speech conversion model is the acoustic features of the sample speech; after receiving the speech conversion instruction, speech conversion is performed on the sentence to be converted according to the acoustic features of the target object to obtain speech that meets the features of the target object, thereby solving the problem of low naturalness of speech conversion and achieving the effect of improving the naturalness of speech conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0020] Figure 1 is a flow chart of a method for processing speech data according to an embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of a speech data processing device according to an embodiment of the present invention;

[0022] Figure 3 Schematic diagram of the device of this embodiment. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] An embodiment of the present invention provides a method for processing voice data.

[0027] Figure 1 is a flow chart of a method for processing speech data according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0028] Step S102: Acquire a phoneme sequence of a sample speech of a target object;

[0029] Step S104: Encode the phoneme sequence according to a preset rule to obtain a phoneme encoding sequence;

[0030] Step S106: Recognize the acoustic features of the sample speech through a preset speech conversion model to obtain the acoustic features of the target object, wherein the input of the speech conversion model is a phoneme coding sequence, and the output of the speech conversion model is the acoustic features of the sample speech;

[0031] Step S108: after receiving the speech conversion instruction, speech conversion is performed on the sentence to be converted according to the acoustic characteristics of the target object to obtain speech that meets the characteristics of the target object.

[0032] This embodiment adopts the method of obtaining a phoneme sequence of a sample speech of a target object; encoding the phoneme sequence according to a preset rule to obtain a phoneme coding sequence; recognizing the acoustic features of the sample speech through a preset speech conversion model to obtain the acoustic features of the target object, wherein the input of the speech conversion model is the phoneme coding sequence, and the output of the speech conversion model is the acoustic features of the sample speech; after receiving a speech conversion instruction, performing speech conversion on the sentence to be converted according to the acoustic features of the target object to obtain a speech that meets the features of the target object, thereby solving the problem of low naturalness of speech conversion, and through the non-equal length mapping of the phoneme coding sequence to the acoustic features, it can be unaffected by the phoneme length of the original sentence, thereby achieving the effect of improving the naturalness of speech conversion.

[0033] In this embodiment, the target object is the object whose voice is ultimately to be imitated, that is, the timbre of a sentence is converted from the original speaker to Zhang San, and spoken in the tone of Zhang San. Before this, it is necessary to obtain the timbre and speaking habits of Zhang San's daily speech. The sample voice is the sample voice of Zhang San's daily speech. The phoneme sequence is obtained and then encoded. The acoustic features are recognized by a preset model to obtain the acoustic features of Zhang San's speech. After receiving the instruction to convert to Zhang San's speech, the voice conversion is performed according to the features of Zhang San's speech extracted from the sample voice to obtain a voice that meets Zhang San's features, and the voice conversion is completed.

[0034] As an optional implementation, the acoustic features of the sample speech are identified by a preset speech conversion model, and the acoustic features of the target object are obtained, including: vector encoding the phoneme coding sequence of the sample speech by the encoder layer of the preset speech conversion model to obtain the coding vector; calculating the alignment similarity between the coding vector and the vector corresponding to the acoustic feature by the decoder layer; obtaining the mapping between the coding vector and the vector corresponding to the acoustic feature when the alignment similarity meets the preset conditions; determining the mapping rule between the coding vector and the vector corresponding to the acoustic feature as the acoustic feature of the target object. Among them, the preset condition can be that the alignment similarity is greater than a certain value, or within a certain value range, and the preset condition can be set and modified according to the needs of the scene.

[0035] As an optional implementation, the length of the encoding vector is equal to the length of the phoneme encoding sequence, and the length of the output vector of the decoder layer is equal to the length of the sample speech.

[0036] The speech conversion model can be an end-to-end deep neural network that can predict acoustic features. It includes an encoder and a decoder. It should be noted that the decoder is a decoder with an attention mechanism. The main function of the encoder is to encode the input language information and use a high-dimensional vector to represent the language information contained in the sentence. The length of the vector is still the same as the phoneme encoding sequence, and has nothing to do with the length of the speech time. The decoder with an attention mechanism calculates the alignment similarity between the high-dimensional language vector and the acoustic feature vector after encoding each frame, and completes the mapping of the high-dimensional language vector to the acoustic feature. The output of the decoder is the same length as the speech length. Therefore, the model as a whole realizes the non-equal length mapping between phoneme encoding and acoustic features.

[0037] As an optional implementation, after receiving a speech conversion instruction, the sentence to be converted is speech-converted according to the acoustic features of the target object to obtain speech that meets the features of the target object, including: extracting a phoneme sequence from the sentence to be converted and then encoding it to obtain a phoneme coding sequence of the sentence to be converted; inputting the phoneme coding sequence of the sentence to be converted into a trained speech conversion model to obtain features that meet the target object, wherein the trained speech conversion model is obtained based on a preset speech conversion model, the input of the trained speech conversion model is a phoneme coding sequence, and the output of the trained speech conversion model is the language habit features of the target object; and performing waveform restoration on the features that meet the target object through an LPCNet vocoder to obtain speech that meets the features of the target object.

[0038] The trained speech conversion model is trained based on the preset speech conversion model to adapt the parameters to the specific target object. The preset speech conversion model can recognize the acoustic characteristics of the speaker from a speech, and the trained speech conversion model can remember Zhang San's acoustic characteristics (language habits). When Li Si speaks a paragraph, it recognizes the content and gives the acoustic characteristics of Zhang San when he speaks this paragraph. After the waveform is restored, the paragraph spoken in Zhang San's tone is obtained.

[0039] Speech reconstruction process: Use LPCNetp[] to restore the 20-dimensional acoustic features to a waveform. LPCNet is a signal and network combined vocoder that can complete speech conversion in real time, and the quality of speech conversion is much higher than that of traditional signal vocoders.

[0040] As an optional implementation, after obtaining the voice that meets the characteristics of the target object, the voice that meets the characteristics of the target object is played on a preset device. For example, the converted voice can be played on a mobile phone terminal or on an audio device.

[0041] As an optional implementation, the acoustic features of the target object are 20-dimensional acoustic features, including 18-dimensional Bark cepstral coefficients, 1-dimensional fundamental frequency inverse, and 1-dimensional correlation coefficient. With the development of technology, the acoustic features of the target object can be more dimensional or less dimensional.

[0042] This embodiment also provides a specific implementation method, the specific contents are as follows:

[0043] The technical solution of this embodiment can be used as a speech conversion method based on non-parallel data and sequence-sequence non-equal length mapping, which can get rid of the dependence on parallel data and effectively improve the naturalness of the converted speech in the speech conversion system, as well as the similarity between the converted speech and the target speaker.

[0044] By extracting the acoustic features of the same sentence of the original speaker and the target speaker, the acoustic features of the two sequences are mapped using the powerful nonlinear modeling capabilities of the neural network (without considering the content information). Therefore, the modeled network can complete the mapping of the acoustic features of the original speaker to the acoustic features of the target speaker, and then restore the acoustic features to speech, that is, complete the speech conversion from the original speaker to the target speaker.

[0045] ASR-TTS (Automatic Speech Recognition- Text to Speech) is a mapping conversion between phoneme sequences and acoustic features based on speech recognition. Its main advantages are: (1) A single phoneme sequence does not contain the pronunciation duration information of the phonemes, so it can be completely modeled based on the prosodic features of the target speaker, making the converted speech more similar to the target speaker; (2) The acoustic posterior probability gives a predicted probability for each frame of the acoustic feature, so it is more susceptible to interference from the original speaker's accent, environmental noise, etc. and has large fluctuations, making the converted speech unclear and impaired in naturalness; while the phoneme sequence uses the language prior knowledge in speech recognition to completely determine the distribution of the output phonemes, thereby greatly reducing the uncertainty of pronunciation and improving the naturalness of the converted speech.

[0046] The implementation of the algorithm and system designed in this embodiment includes two stages:

[0047] 1. Training phase: includes feature preprocessing and model training.

[0048] Feature preprocessing:

[0049] (1) Language feature extraction: Use a pre-trained speech recognition system to extract non-repeating phoneme sequences from the training set. For example, the content of sentence 1 is "The weather is very good today." The phoneme sequence extracted by the recognition system is "j in1 tian1 t ian1 qi4 h en3 h ao3". This sequence has nothing to do with the time length of the original speech and only corresponds to its language content.

[0050] (2) Mapping phoneme sequences into phoneme codes: The types of phonemes modeled by the speech recognition system are limited, and the directly recognized phoneme sequences are strings that cannot be directly used as input to the speech conversion system. Therefore, the phoneme sequences are encoded into preset phoneme numbers. For example, the phoneme sequence of “j in1 t ian1 t ian1 qi4 h en3 h ao3” is mapped to “214 140236 36 236 36 226 110 212 90 212 50”. This sequence is referred to as the phoneme code below.

[0051] (3) Acoustic feature extraction: 20-dimensional acoustic features are extracted from the training set speech, including 18-dimensional Bark cepstral coefficients, 1-dimensional fundamental frequency inverse, and 1-dimensional correlation coefficient.

[0052] Model training:

[0053] The speech conversion model used in this embodiment is an end-to-end deep neural network. The input of the network is the phoneme coding sequence obtained by speech recognition, and the prediction target is the 20-dimensional acoustic features mentioned above. The end-to-end deep neural network includes two parts: an encoder and a decoder with an attention mechanism. The main function of the encoder is to encode the input language information, and use a high-dimensional vector to represent the language information contained in the sentence. The length of the vector is still the same as the phoneme coding sequence, and has nothing to do with the length of the speech time; the decoder with an attention mechanism calculates the alignment similarity between the high-dimensional language vector and the acoustic feature vector after each frame of encoding, and completes the mapping of the high-dimensional language vector to the acoustic feature. The output of the decoder is the same length as the speech length. Thus, the model as a whole realizes the non-equal-length mapping between phoneme coding and acoustic features.

[0054] The loss function of the model is expressed as,

[0055] Where Yi represents the i-th frame of the acoustic feature sequence, Yi pred Represents the acoustic feature sequence of the i-th frame predicted by the speech conversion model, and the speech contains N frames of features.

[0056] Because phoneme encoding completely strips away the phoneme prosodic information in a sentence, the model can better model the prosodic information of the target speaker based on the target speaker’s training data.

[0057] The parameters of the speech conversion model are as follows:

[0058] Table 1: Network structure and parameters

[0059] Pre-net FC-256-Relu-Dropout(0.5)FC-128-Relu-Dropout(0.5) encoder Conv1d blocks: K=16, conv-k-128- ReluMax Poolong: stride=1,width=2Conv1dProjection: conv-3-128-Relu, conv-3-128-LinearHighway net:4-layer-FC-128-ReluBidrectional GRU: 128 cells Attention-based decoder Attention RNN: 1-layer-256-GRUDecoder RNN: 2-layer-256-residual-GRUDense-20-cells

[0060] FC stands for fully connected layer, GRU stands for gated recurrent neural network, and RNN stands for recurrent neural network.

[0061] 2. Prediction stage: It includes three parts: feature phoneme encoding feature preparation, model prediction and speech reconstruction.

[0062] (1) Preparation of phoneme coding features: Use the above-mentioned speech recognition system to extract phoneme sequences from the original sentences and convert them into phoneme coding sequences;

[0063] (2) Model prediction: Input the phoneme encoding sequence into the trained speech conversion model to predict the 20-dimensional acoustic features;

[0064] (3) Speech reconstruction: Use LPCNetp[] to restore the 20-dimensional acoustic features into waveforms. LPCNet is a signal and network combined vocoder that can complete speech conversion in real time, and the quality of speech conversion is much higher than that of traditional signal vocoders.

[0065] In this embodiment, the original speech and the converted audio implement non-equal length mapping, so that the converted speech is not interfered by the phoneme prosodic information in the original sentence, and is closer to the pronunciation style of the target speaker, so that the speech conversion technology gets rid of the dependence on parallel data, greatly reduces the threshold and training cost of speech conversion technology, and has a higher conversion similarity.

[0066] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0067] An embodiment of the present invention provides a speech data processing device, which can be used to execute the speech data processing method of the embodiment of the present invention.

[0068] Figure 2 is a schematic diagram of a speech data processing device according to an embodiment of the present invention. Figure 2 As shown, the device comprises:

[0069] An acquisition unit 10 is used to acquire a phoneme sequence of a sample speech of a target object;

[0070] The encoding unit 20 is used to encode the phoneme sequence according to a preset rule to obtain a phoneme encoding sequence;

[0071] The recognition unit 30 is used to recognize the acoustic features of the sample speech through a preset speech conversion model to obtain the acoustic features of the target object, wherein the input of the speech conversion model is a phoneme coding sequence, and the output of the speech conversion model is the acoustic features of the sample speech;

[0072] The conversion unit 40 is used to perform voice conversion on the sentence to be converted according to the acoustic characteristics of the target object after receiving the voice conversion instruction, so as to obtain a voice that meets the characteristics of the target object.

[0073] This embodiment uses an acquisition unit 10 for acquiring a phoneme sequence of a sample speech of a target object; an encoding unit 20 for encoding the phoneme sequence according to a preset rule to obtain a phoneme coding sequence; a recognition unit 30 for recognizing the acoustic features of the sample speech through a preset speech conversion model to obtain the acoustic features of the target object, wherein the input of the speech conversion model is the phoneme coding sequence, and the output of the speech conversion model is the acoustic features of the sample speech; and a conversion unit 40 for performing speech conversion on the sentence to be converted according to the acoustic features of the target object after receiving a speech conversion instruction to obtain a speech that meets the features of the target object. This solves the problem of low naturalness of speech conversion, and further achieves the effect of improving the naturalness of speech conversion.

[0074] Optionally, the recognition unit 30 includes: an encoding module, which is used to vector encode the phoneme encoding sequence of the sample speech through the encoder layer of a preset speech conversion model to obtain a coding vector; a calculation module, which is used to calculate the alignment similarity between the coding vector and the vector corresponding to the acoustic feature through the decoder layer; a mapping module, which is used to obtain the mapping between the coding vector and the vector corresponding to the acoustic feature when the alignment similarity meets the preset conditions; and a determination module, which is used to determine the mapping rule between the coding vector and the vector corresponding to the acoustic feature as the acoustic feature of the target object.

[0075] The speech data processing device includes a processor and a memory. The acquisition unit, encoding unit and recognition unit are stored in the memory as program units. The processor executes the program units stored in the memory to realize corresponding functions.

[0076] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the naturalness of the speech conversion can be improved by adjusting the kernel parameters.

[0077] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0078] An embodiment of the present invention provides a storage medium on which a program is stored. When the program is executed by a processor, the voice data processing method is implemented.

[0079] An embodiment of the present invention provides an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the voice data processing method of the present invention.

[0080] An embodiment of the present invention provides a device, Figure 3is a schematic diagram of a device of this embodiment, the device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the above-mentioned voice data processing method. The device in this article can be a server, PC, PAD, mobile phone, etc.

[0081] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: obtaining a phoneme sequence of a sample speech of a target object; encoding the phoneme sequence according to preset rules to obtain a phoneme coding sequence; identifying the acoustic features of the sample speech through a preset speech conversion model to obtain the acoustic features of the target object, wherein the input of the speech conversion model is the phoneme coding sequence, and the output of the speech conversion model is the acoustic features of the sample speech; after receiving a speech conversion instruction, performing speech conversion on the sentence to be converted according to the acoustic features of the target object to obtain speech that meets the features of the target object.

[0082] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0083] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0084] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0086] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0087] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0088] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0089] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0090] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0091] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for processing speech data, characterized in that: include: Obtaining a phoneme sequence of a sample speech of a target object; Encoding the phoneme sequence according to a preset rule to obtain a phoneme encoding sequence; The acoustic features of the sample speech are identified by a preset speech conversion model to obtain the acoustic features of the target object, including: vector encoding the phoneme coding sequence of the sample speech by the encoder layer of the preset speech conversion model to obtain the coding vector, calculating the alignment similarity between the coding vector and the vector corresponding to the acoustic feature by the decoder layer, and obtaining the mapping between the coding vector and the vector corresponding to the acoustic feature when the alignment similarity meets the preset conditions, and determining the mapping rule between the coding vector and the vector corresponding to the acoustic feature as the acoustic feature of the target object, wherein the input of the speech conversion model is the phoneme coding sequence, and the output of the speech conversion model is the acoustic features of the sample speech. The decoder layer is a decoder with an attention mechanism, the length of the encoding vector is equal to the length of the phoneme encoding sequence, the length of the output vector of the decoder layer is equal to the length of the sample speech, the speech conversion model is an end-to-end deep neural network for predicting acoustic features, including an encoder and a decoder, the encoder encodes the input language information, and uses a high-dimensional vector to represent the language information contained in the sentence, the length of the vector is equal to the length of the phoneme encoding sequence, and has nothing to do with the speech time length; the decoder with an attention mechanism calculates the alignment similarity between the high-dimensional language vector and the acoustic feature vector after each frame of encoding, and completes the mapping of the high-dimensional language vector to the acoustic feature, and the preset conditions at least include: the alignment similarity is greater than a preset value; After receiving a speech conversion instruction, speech conversion is performed on the sentence to be converted according to the acoustic features of the target object to obtain speech that meets the features of the target object, including: extracting a phoneme sequence from the sentence to be converted and then encoding it to obtain a phoneme coding sequence of the sentence to be converted, inputting the phoneme coding sequence of the sentence to be converted into a trained speech conversion model to obtain features that meet the target object, wherein the trained speech conversion model is obtained based on the preset speech conversion model, the input of the trained speech conversion model is a phoneme coding sequence, the output of the trained speech conversion model is the language habit features of the target object, and waveform recovery is performed on the features that meet the target object to obtain speech that meets the features of the target object.

2. The method according to claim 1, characterized in that Performing waveform recovery on the features that match the target object to obtain speech that matches the features of the target object includes: The waveform of the feature that meets the target object is restored by an LPCNet vocoder to obtain a speech that meets the feature of the target object.

3. The method according to claim 1, characterized in that After obtaining the speech that meets the characteristics of the target object, the method further includes: The voice matching the characteristics of the target object is played on a preset device.

4. The method according to claim 1, characterized in that: The acoustic features of the target object are 20-dimensional acoustic features, including 18-dimensional Bark cepstral coefficients, 1-dimensional fundamental frequency inverse, and 1-dimensional correlation coefficient.

5. A voice data processing device, characterized in that: include: An acquisition unit, used for acquiring a phoneme sequence of a sample speech of a target object; An encoding unit, used for encoding the phoneme sequence according to a preset rule to obtain a phoneme encoding sequence; The recognition unit is used to recognize the acoustic features of the sample speech through a preset speech conversion model to obtain the acoustic features of the target object, including: an encoding module, used to vector encode the phoneme encoding sequence of the sample speech through the encoder layer of the preset speech conversion model to obtain the encoding vector, a calculation module, used to calculate the alignment similarity between the encoding vector and the vector corresponding to the acoustic feature through the decoder layer, a mapping module, used to obtain the mapping between the encoding vector and the vector corresponding to the acoustic feature when the alignment similarity meets the preset conditions, and a determination module, used to determine the mapping rule between the encoding vector and the vector corresponding to the acoustic feature as the acoustic feature of the target object, wherein the input of the speech conversion model is the phoneme encoding sequence, and the speech The output of the conversion model is the acoustic feature of the sample speech, the decoder layer is a decoder with an attention mechanism, the length of the encoding vector is equal to the length of the phoneme encoding sequence, the length of the output vector of the decoder layer is equal to the length of the sample speech, the speech conversion model is an end-to-end deep neural network for predicting acoustic features, including an encoder and a decoder, the encoder encodes the input language information, and uses a high-dimensional vector to represent the language information contained in the sentence, the length of the vector is equal to the length of the phoneme encoding sequence, and is independent of the speech time length; the decoder with an attention mechanism calculates the alignment similarity between the high-dimensional language vector and the acoustic feature vector after each frame of encoding, and completes the mapping of the high-dimensional language vector to the acoustic feature, and the preset conditions at least include: the alignment similarity is greater than a preset value; A conversion unit is used to perform voice conversion on a sentence to be converted according to the acoustic features of the target object after receiving a voice conversion instruction to obtain a voice that meets the features of the target object, including: extracting a phoneme sequence from the sentence to be converted and then encoding it to obtain a phoneme coding sequence of the sentence to be converted, inputting the phoneme coding sequence of the sentence to be converted into a trained voice conversion model to obtain the features that meet the target object, wherein the trained voice conversion model is obtained based on the preset voice conversion model, the input of the trained voice conversion model is the phoneme coding sequence, the output of the trained voice conversion model is the language habit features of the target object, and waveform recovery is performed on the features that meet the target object to obtain a voice that meets the features of the target object.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed by a processor, the speech data processing method according to any one of claims 1 to 4 is performed.

7. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the voice data processing method as described in any one of claims 1 to 4.

Images