Voice playback method, device, storage medium and electronic device

Abstract

The present disclosure relates to a speech playback method, device, storage medium and electronic device, the method comprising: encoding each phoneme vector in a phoneme vector sequence corresponding to a text to be synthesized to obtain a coding vector sequence; predicting the coding vector sequence for speech information to obtain speech information corresponding to each coding vector in the coding vector sequence; determining each decoding group and a decoding order corresponding to each decoding group based on the sequence of arrangement of each coding vector in the coding vector sequence and the decoding efficiency; decoding each decoding group based on the decoding order; in response to obtaining the first speech data through decoding, starting to play the decoded speech data in the order of the decoded speech data. The method disclosed in the present disclosure can shorten the waiting time for speech playback and improve the efficiency of human-computer communication while ensuring the audio quality.

Description

Technical Field

[0001] The present disclosure relates to the field of voice technology, and more specifically, to a voice playback method, device, storage medium, and electronic device. Background Art

[0002] With the development of deep learning in recent years, speech playback based on speech synthesis technology has also been greatly improved. Speech playback based on speech synthesis technology refers to converting input text into natural and fluent speech, so that the machine can speak, expanding the human-computer interaction mode and making human-computer communication more convenient.

[0003] However, in the speech playback method based on speech synthesis technology in the related art, when the text data of a single request is too large, for example, dozens or hundreds of words, the waiting time from inputting the requested text data to playing the voice is long, which reduces the efficiency of human-computer communication. Summary of the invention

[0004] The purpose of the present disclosure is to provide a voice playback method, device, storage medium and electronic device, which solves the problem that the waiting time from inputting a text to be synthesized to playing the voice is long, thereby reducing the efficiency of human-computer communication.

[0005] In order to achieve the above-mentioned object, in a first aspect, the present disclosure provides a voice playback method, the method comprising:

[0006] Encoding each phoneme vector in the phoneme vector sequence corresponding to the text to be synthesized to obtain an encoding vector sequence;

[0007] Performing prediction processing on the coding vector sequence for speech information to obtain speech information corresponding to each coding vector in the coding vector sequence;

[0008] Determine each decoding group and a decoding order corresponding to each decoding group based on the sequence of arrangement of each encoding vector in the encoding vector sequence and the decoding efficiency, wherein the decoding group includes at least one encoding vector and its corresponding voice information, and the decoding efficiency represents the number of encoding vectors to be decoded each time;

[0009] Based on the decoding order, decoding each of the decoding packets;

[0010] In response to the first voice data being obtained through decoding, the decoded voice data begins to be played in the order in which the voice data is obtained through decoding.

[0011] In some embodiments, the method further comprises:

[0012] The decoding efficiency is determined based on a decoding efficiency configuration parameter, wherein the decoding efficiency configuration parameter is determined based on an acceptance level of a voice playback delay.

[0013] In some embodiments, the method further comprises:

[0014] Performing phoneme extraction on the text to be synthesized to obtain a phoneme sequence corresponding to the text to be synthesized;

[0015] Perform phoneme embedding processing on the phoneme sequence to obtain the phoneme vector sequence.

[0016] In some implementations, the voice information includes a combination of one or more of duration, pitch, and volume.

[0017] In some implementations, encoding each phoneme vector in the phoneme vector sequence includes:

[0018] Encoding each phoneme vector in the phoneme vector sequence by an encoder;

[0019] The performing prediction processing of speech information on the coding vector sequence comprises:

[0020] Performing speech information prediction processing on the coding vector sequence by a speech information predictor;

[0021] The decoding of each decoding packet in sequence based on the decoding order includes:

[0022] Based on the decoding order, decoding the respective decoding packets in sequence by a decoder;

[0023] The encoder, speech information predictor and decoder are obtained through joint training based on sample speech data carrying text annotations.

[0024] In some embodiments, the method further comprises:

[0025] Acquire interactive speech in the target environment;

[0026] Converting the interactive speech into a corresponding first interactive text;

[0027] Acquire a first reply text corresponding to the first interactive text, and use the first reply text as the text to be synthesized; or

[0028] Obtaining a second interactive text input by the user;

[0029] A second reply text corresponding to the second interactive text is obtained, and the second reply text serves as the text to be synthesized.

[0030] In a second aspect, the present disclosure further provides a voice playback device, the device comprising:

[0031] An encoding module, used for encoding each phoneme vector in the phoneme vector sequence corresponding to the text to be synthesized to obtain an encoding vector sequence;

[0032] A prediction module, used to perform prediction processing on the coding vector sequence for speech information to obtain speech information corresponding to each coding vector in the coding vector sequence;

[0033] A grouping module, configured to determine each decoding group and a decoding order corresponding to each decoding group based on the order in which each encoding vector in the encoding vector sequence is arranged and the decoding efficiency, wherein the decoding group includes at least one encoding vector and its corresponding speech information, and the decoding efficiency represents the number of encoding vectors to be decoded each time;

[0034] A decoding module, configured to decode each of the decoding packets based on the decoding order;

[0035] The playing module is used for starting to play the decoded voice data in response to the first voice data obtained through decoding in the order in which the voice data is obtained through decoding.

[0036] In a third aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method in the first aspect when executed by a processor.

[0037] In a fourth aspect, the present disclosure further provides an electronic device, including:

[0038] a memory having a computer program stored thereon;

[0039] A processor is used to execute the computer program in the memory to implement the steps of the method in the first aspect.

[0040] Through the above technical solution, since the vector of a decoding group is decoded each time and the voice data is played in the order of decoding, the amount of data processed simultaneously is reduced, the decoding load is reduced, and the decoding time of a single decoding is reduced, so that the time from inputting text to the first voice playback is reduced, and the user can hear the sound faster. Therefore, while ensuring the audio quality, the waiting time for voice playback is shortened and the efficiency of human-computer communication is improved.

[0041] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0043] Figure 1 It is a flowchart of a voice playback method provided by an embodiment of the present disclosure.

[0044] Figure 2 It is a flowchart of another voice playback method provided by an embodiment of the present disclosure.

[0045] Figure 3 It is a structural diagram of a deep neural network model provided by an embodiment of the present disclosure.

[0046] Figure 4 It is a structural schematic diagram of a voice playback device provided by an embodiment of the present disclosure.

[0047] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0049] In the related art, in order to ensure audio quality and improve speech synthesis speed, there is a non-autoregressive speech synthesis model FastSpeech2. However, the inventors found in actual projects that the speech synthesis speed of FastSpeech2 is positively correlated with the amount of text data requested for synthesis, that is, when the amount of text data requested for synthesis is small, it can have a faster speech synthesis speed, so that the time from inputting the requested text data to playing the speech can meet the user's needs. In the case of a large amount of text data requested in a single request, for example, dozens of words, hundreds of words or even more, the waiting time from inputting the requested text data to playing the speech is long, up to several seconds or even longer, which can no longer meet the user's needs and reduce the efficiency of human-computer communication.

[0050] In order to solve the above problems, the inventors proposed the voice playback method, device, storage medium and electronic device provided by the present invention, firstly, each phoneme vector in the phoneme vector sequence corresponding to the text to be synthesized is encoded to obtain a coding vector sequence; then, the coding vector sequence is predicted to obtain the voice information corresponding to each coding vector in the coding vector sequence; then, based on the order of arrangement of each coding vector in the coding vector sequence and the decoding efficiency, each decoding group and the decoding order corresponding to each decoding group are determined; then, based on the decoding order, each decoding group is decoded; then, in response to the first voice data obtained by decoding, the decoded voice data is played in the order of the decoded voice data.

[0051] Since the vector of a decoding group is decoded each time and the voice data is played in the order obtained by decoding, the amount of data processed simultaneously is reduced, the decoding load is reduced, and the decoding time of a single decoding is reduced, so that the time from inputting text to the first voice playback is reduced, and the user can hear the sound faster. In addition, the playback of voice data starts from the decoding to obtain the first voice data, and the voice playback itself also takes time, and the playback time of the voice data corresponding to a decoding group is usually greater than the decoding time of the decoding group, so that the decoding can continue within the time of voice output, so there is no need to consider the total decoding time. Therefore, the above method can shorten the waiting time of voice playback while ensuring the audio quality, and improve the efficiency of human-computer communication.

[0052] Before further describing the embodiments of the present disclosure in detail, some exemplary application scenarios involved in the embodiments of the present disclosure are introduced.

[0053] The application scenario can be a human-computer interaction scenario. In an exemplary human-computer interaction scenario, the user inputs interactive voice, and after receiving the interactive voice, the terminal device can first convert the interactive voice into corresponding interactive text, and then obtain the corresponding reply text based on the interactive text. After obtaining the reply text, the terminal device can use the reply text as the text to be synthesized, and then process the text to be synthesized to obtain voice data and play it, thereby completing the entire human-computer interaction process.

[0054] In another exemplary human-computer interaction scenario, the user directly inputs the interaction text, for example, through buttons or touch screens. After receiving the interaction text, the terminal device can obtain the corresponding reply text based on the interaction text. After obtaining the reply text, the terminal device can use the reply text as the text to be synthesized, and then process the text to be synthesized to obtain voice data and play it, thereby completing the entire human-computer interaction process.

[0055] See also Figure 1 , Figure 1 is a flow chart of a voice playing method according to an exemplary embodiment of the present disclosure, the method is applied to a terminal device, the terminal device may be an intelligent voice robot, a mobile phone, a notebook, a tablet computer, a desktop computer, a smart speaker, a smart TV, a wearable device, and a vehicle terminal, etc. Figure 1 As shown, the voice playing method includes steps S110 to S150.

[0056] Specific:

[0057] S110, encoding each phoneme vector in the phoneme vector sequence corresponding to the text to be synthesized to obtain an encoding vector sequence.

[0058] The text to be synthesized refers to text used for synthesizing speech. The text to be synthesized may correspond to a phoneme vector sequence, and each phoneme vector sequence includes multiple phoneme vectors.

[0059] Among them, phonemes are the smallest units of speech divided according to the natural properties of speech. From the perspective of acoustic properties, phonemes are the smallest units of speech divided from the perspective of sound quality. From the perspective of physiological properties, one pronunciation action forms a phoneme. For example, "ma" contains two pronunciation actions, "m" and "a", which are two phonemes. The sound produced by the same pronunciation action is the same phoneme, and the sounds produced by different pronunciation actions are different phonemes. For example, in "ma-mi", the two "m" have the same pronunciation action, which is the same phoneme, and the pronunciation actions of "a" and "i" are different, which are different phonemes.

[0060] Each phoneme vector in the phoneme vector sequence corresponding to the text to be synthesized is a vector representation of a low-dimensional space, and the language meaning features that can be expressed are limited. Therefore, in order to facilitate the subsequent speech synthesis and playback of the phoneme vector sequence corresponding to the text to be synthesized, and to ensure the voice quality, each phoneme vector in the phoneme vector sequence corresponding to the text to be synthesized can be encoded to obtain a coding vector sequence. Each coding vector in the coding vector sequence is a vector representation of a high-dimensional space, which can express more and more abstract language meaning features. It can be understood that the number of vectors in the coding vector sequence is the same as the number of vectors in the phoneme vector sequence, except that the dimension of each vector is different. In addition, in the coding vector sequence and in the phoneme vector sequence, the order of arrangement of each vector relative to each other remains unchanged. For example, the first phoneme vector in the phoneme vector sequence is still the first vector arranged in the coding vector sequence after encoding, and the Nth phoneme vector in the phoneme vector sequence is still the Nth vector arranged in the coding vector sequence after encoding.

[0061] In some embodiments, encoding processing of each phoneme vector in a phoneme vector sequence corresponding to the text to be synthesized may include at least one of the following operations: performing addition operations on each phoneme vector, performing linear transformation operations on each phoneme vector, performing nonlinear transformation operations on each phoneme vector, and performing dot multiplication operations on each phoneme vector.

[0062] S120, performing speech information prediction processing on the coding vector sequence to obtain speech information corresponding to each coding vector in the coding vector sequence.

[0063] The voice information may include one or more of duration, pitch and volume.

[0064] In the disclosed embodiment, after the coding vector sequence is processed for prediction of speech information, corresponding speech information can be obtained for each coding vector.

[0065] In the disclosed embodiment, since the speech information is predicted and processed for the entire coding vector sequence, the predicted speech information can carry context information in the text to be synthesized, thereby improving the quality of the subsequently played speech.

[0066] For example, assuming that the text to be synthesized is "You are really studious. I should study hard like you", in the obtained voice information, it can be found that the three phonemes "hao" included in the first word "good" are shorter in duration and louder in volume than the three phonemes "hao" included in the second and third words "good" respectively. Therefore, the contextual information can be reflected from the voice information.

[0067] S130, based on the order of arrangement of each coding vector in the coding vector sequence and the decoding efficiency, determine each decoding group and the decoding order corresponding to each decoding group, the decoding group includes at least one coding vector and its corresponding voice information, and the decoding efficiency represents the number of coding vectors decoded each time.

[0068] It can be understood that in the above step S120, the speech information prediction processing is performed on the entire coding vector sequence at the same time, and in step S130, the various coding vectors in the coding vector sequence are grouped according to the order of arrangement in the coding vector sequence and the decoding efficiency, so that each decoding group can be decoded in turn in the subsequent steps.

[0069] Exemplarily, assuming that there are 10 coding vectors (vector 1, vector 2, vector 3, ..., vector 10) in sequence in the coding vector sequence, and the decoding efficiency is decoding 3 coding vectors each time, at this time, it can be determined that there are 4 decoding groups in total, namely the decoding group located in the first decoding order (vector 1 and the corresponding voice information, vector 2 and the corresponding voice information, vector 3 and the corresponding voice information), the decoding group located in the second decoding order (vector 4 and the corresponding voice information, vector 5 and the corresponding voice information, vector 6 and the corresponding voice information), the decoding group located in the third decoding order (vector 7 and the corresponding voice information, vector 8 and the corresponding voice information, vector 9 and the corresponding voice information) and the decoding group located in the fourth decoding order (vector 10 and the corresponding voice information).

[0070] In some implementations, the decoding efficiency may be determined based on a decoding efficiency configuration parameter, wherein the decoding efficiency configuration parameter is determined based on an acceptance level of a voice playback delay.

[0071] In the disclosed embodiment, the user can set the decoding efficiency configuration parameters corresponding to the electronic device according to the user's acceptance of the voice playback delay, so that the electronic device can determine the decoding efficiency based on the decoding efficiency configuration parameters.

[0072] It is understandable that the greater the degree of acceptance of voice playback delay, the higher the decoding efficiency corresponding to the decoding efficiency configuration parameter, that is, the larger the number of coding vectors decoded each time. It should be noted that the higher the decoding efficiency, the longer the single decoding time.

[0073] In addition, it should be noted that the higher the decoding efficiency, the fewer the number of voice data segments obtained, which can relatively reduce the number of memory reads and save power consumption. The lower the decoding efficiency, the more the number of voice data segments obtained, which relatively increases the number of memory reads and increases power consumption. For example, if the encoding vector corresponding to each phoneme and the voice information corresponding to the encoding vector are taken as a decoding group, when the number of texts is in the order of 100, assuming that there are 300 phonemes, in this case, the storage and reading of voice data in this human-computer interaction process will read the memory 300 times, thereby increasing power consumption. If the encoding vectors corresponding to 5 phonemes and the voice information corresponding to these 5 encoding vectors are taken as a decoding group, the memory can be read 60 times, thereby reducing power consumption.

[0074] Therefore, in some implementations, the decoding efficiency can be appropriately increased within the user's acceptance range of voice playback delay.

[0075] S140, decoding each decoding group in sequence based on the decoding order.

[0076] Among them, decoding each decoding packet in sequence based on the decoding order means decoding one decoding packet each time, and decoding the next decoding packet after the previous decoding packet is decoded.

[0077] Continuing with the above example, the decoding group in the first decoding order (vector 1 and the corresponding voice information, vector 2 and the corresponding voice information, vector 3 and the corresponding voice information) is decoded first, and after decoding to obtain a voice data, the decoding group in the second decoding order (vector 4 and the corresponding voice information, vector 5 and the corresponding voice information, vector 6 and the corresponding voice information) is decoded to obtain another voice data, and so on.

[0078] S150, in response to obtaining the first voice data through decoding, starting to play the decoded voice data in the order in which the voice data is decoded.

[0079] In the disclosed embodiment, after decoding the decoding group located in the first decoding order to obtain the first voice data, the decoded voice data is immediately played in the order in which the voice data is decoded, thereby eliminating the need to wait until all the decoding groups are decoded before starting to play the voice data.

[0080] In addition, it can be understood that since the individual coding vectors in the coding vector sequence are grouped, the number of coding vectors in each decoding group is relatively small. In this way, the playback time of the voice data corresponding to a certain decoding group is usually greater than the decoding time of the decoding group. That is to say, in the process of playing the first decoded voice data, the second decoded voice data is already prepared. After the first voice data is played, the second voice data can be played continuously, thereby ensuring smooth voice playback and further ensuring voice quality.

[0081] In some implementations, a vocoder may be used to play the voice data.

[0082] It can be understood that the decoding process is a more complicated process in speech synthesis. Compared with the encoding process and the prediction process of speech information, a large amount of time loss in the entire speech synthesis is caused by this process. In the embodiment of the present disclosure, since all the encoding vectors in the encoding vector sequence are no longer decoded at one time, and the speech data is not played after all the decoding is completed, but the vectors of a decoding group are decoded each time, and the speech data are played in the order obtained by decoding, the amount of data processed at the same time is reduced, the decoding load is reduced, and the decoding time of a single decoding is reduced, so that the time from inputting text to playing the speech for the first time is reduced, and the user The sound can also be heard faster. In addition, although only one decoding group is decoded each time, the total decoding time of multiple decodings may be greater than the total decoding time of all decodings at one time. However, in the embodiment of the present disclosure, the playback of voice data starts from the decoding of the first voice data, and the voice playback itself also takes time, and the playback time of the voice data corresponding to a certain decoding group is usually greater than the decoding time of the decoding group. In this way, decoding can continue within the time of voice output, so there is no need to consider the total decoding time. Therefore, the above method can shorten the waiting time for voice playback while ensuring the audio quality, thereby improving the efficiency of human-computer communication.

[0083] In combination with the foregoing content, it can be known that the text to be synthesized can be obtained based on the interactive speech. In this case, in some embodiments, the method of the embodiment of the present disclosure may also include the steps of: obtaining the interactive speech in the target environment; converting the interactive speech into a corresponding first interactive text; obtaining a first reply text corresponding to the first interactive text, and the first reply text is used as the text to be synthesized.

[0084] Among them, the target environment refers to the environment in which the terminal device is currently located. After the terminal device turns on the microphone, it can collect voice information in the current environment and obtain interactive voice.

[0085] It is understandable that in some cases, there may only be interactive voice input by the user in the target environment. In this case, the terminal device can directly use the collected voice as the interactive voice. In other cases, there may be both interactive voice input by the user and other noises other than the interactive voice in the target environment. In this case, the terminal device can remove interference from the collected voice, and use the voice after the interference is removed as the interactive voice.

[0086] After obtaining the interactive voice, the terminal device can convert the interactive voice into the corresponding first interactive text, and then obtain the first reply text corresponding to the first interactive text, and then use the first reply text as the text to be synthesized, so as to synthesize the text to be synthesized into voice data for playback.

[0087] The first reply text may be obtained by the server based on the first interactive text and sent by the server to the terminal device, or may be obtained by the terminal device itself based on the first interactive text.

[0088] In addition, the text to be synthesized can be obtained based on the interactive text. In this case, in some other implementations, the method of the embodiment of the present disclosure may also include the steps of: obtaining a second interactive text input by the user; obtaining a second reply text corresponding to the second interactive text, and the second reply text is used as the text to be synthesized.

[0089] Similarly, the second reply text can be obtained by the server according to the second interactive text and sent to the terminal device by the server, or it can be obtained by the terminal device itself according to the second interactive text. Then, after obtaining the second reply text, the terminal device can use the second reply text as the text to be synthesized, so as to synthesize the text to be synthesized into voice data for playback.

[0090] See also Figure 2 , Figure 2 is a flow chart of a voice playing method according to an exemplary embodiment of the present disclosure, the method is applied to a terminal device, the terminal device may be an intelligent voice robot, a mobile phone, a notebook, a tablet computer, a desktop computer, a smart speaker, a smart TV, a wearable device, and a vehicle terminal, etc. Figure 1 As shown, the voice playing method includes steps S210 to S270.

[0091] Specific:

[0092] S210, extracting phonemes from the text to be synthesized to obtain a phoneme sequence corresponding to the text to be synthesized.

[0093] As an implementation method, a phoneme extraction algorithm may be used to extract phonemes from the text to be synthesized, thereby obtaining a phoneme sequence corresponding to the text to be synthesized.

[0094] For example, assuming that the text to be synthesized is "Today is a really good day", after using the phoneme extraction algorithm to extract phonemes from the text to be synthesized, the corresponding phoneme sequence "jintianzhenshig eh aorizi" can be obtained.

[0095] It should be noted that when phoneme extraction is performed on the text to be synthesized, in order to improve the accuracy of the subsequently played speech, phoneme extraction can be performed according to pronunciation rules. For example, if the text to be synthesized is Chinese text, phoneme extraction can be performed based on Chinese phonemes. If the text to be synthesized is English text, phoneme extraction can be performed based on phonetic symbols.

[0096] S220, performing phoneme embedding processing on the phoneme sequence to obtain a phoneme vector sequence.

[0097] There are many ways to perform phoneme embedding processing on a phoneme sequence to obtain a phoneme vector sequence.

[0098] In some implementations, a preset phoneme conversion table may be used to perform phoneme embedding processing on a phoneme sequence to obtain a phoneme vector sequence. In the phoneme conversion table, each phoneme corresponds to a vector representation, so that the phoneme vector sequence corresponding to the phoneme sequence can be obtained by looking up the table.

[0099] In other implementations, a pre-trained neural network model may be used to perform phoneme embedding processing on a phoneme sequence to obtain a phoneme vector sequence.

[0100] S230, encoding each phoneme vector in the phoneme vector sequence corresponding to the text to be synthesized to obtain an encoding vector sequence.

[0101] S240, performing speech information prediction processing on the coding vector sequence to obtain speech information corresponding to each coding vector in the coding vector sequence.

[0102] S250, determining each decoding group and a decoding order corresponding to each decoding group based on the sequence of arrangement of each encoding vector in the encoding vector sequence and the decoding efficiency.

[0103] S260, decoding each decoding group based on the decoding order.

[0104] S270, in response to obtaining the first voice data through decoding, start playing the decoded voice data in the order in which the voice data is obtained through decoding.

[0105] The detailed description of the above steps S230 to S270 can refer to the detailed description of the above steps S110 to S150, which will not be repeated here.

[0106] It should be noted that, in some implementations, steps S230 to S260 in the disclosed embodiment may be performed by a trained deep neural network model, where Figure 3As shown, the deep neural network model may include an encoder, a speech information predictor, a data register and a decoder. In this case, encoding each phoneme vector in a phoneme vector sequence may include the steps of: encoding each phoneme vector in the phoneme vector sequence by an encoder; predicting speech information on the encoded vector sequence may include the steps of: predicting speech information on the encoded vector sequence by a speech information predictor; decoding each decoding group in turn based on a decoding order, may include the steps of: decoding each decoding group in turn by a decoder based on a decoding order.

[0107] The data register may be used to store decoding groups, each decoding group corresponding to a decoding order.

[0108] The encoder, speech information predictor and decoder can be adjusted in parameters by joint training. The joint training means that in each training, the parameters of the encoder, speech information predictor and decoder are adjusted as a whole. In the joint training process, the training samples used are sample speech data with text annotations.

[0109] In some embodiments, Mel spectrum extraction can be performed on the sample speech data to obtain the Mel spectrum of the sample speech data, and then the phoneme vector sequence corresponding to the text annotation corresponding to the sample speech data is input into the deep neural network model, and the predicted Mel spectrum is obtained by sequentially processing the encoder, speech information predictor, data register and decoder in the deep neural network model. Based on the difference between the predicted Mel spectrum and the Mel spectrum of the sample speech data, the parameters of the encoder, speech information predictor and decoder are adjusted as a whole, thereby completing one training of the deep neural network model. Among them, the decoder can be a decoder that decodes to obtain the Mel spectrum, that is, the speech data at this time is the corresponding Mel spectrum.

[0110] In addition, considering that the sample speech data carrying text annotations may be obtained by inputting the sample speech data into a speech recognition model, there may be some meaningless symbols or redundant information in the text annotations. Therefore, in order to improve the accuracy of deep neural network model training, in some embodiments, after obtaining the sample data, the text annotations carried by the sample data can be removed of meaningless symbols and redundant information.

[0111] In combination with the foregoing content, it can be known that in some implementations, the decoding efficiency can be determined based on the decoding efficiency configuration parameters. After the decoding efficiency is determined, the decoding efficiency can be set as a configuration parameter of the data register in the above-mentioned deep neural network model.

[0112] It should be noted that the data register does not participate in training and this parameter will not change during the training process.

[0113] The effect of the voice playback method provided in the embodiment of the present disclosure is described below through test data. In the electronic device, the process from receiving interactive voice or interactive text to obtaining the text to be synthesized is the same and does not need to be tested. Therefore, in the embodiment of the present disclosure, the effect mainly includes two aspects. One is the waiting time from inputting the text to be synthesized to playing the voice, and the other is the quality of the played voice data.

[0114] The experiment was conducted using three groups of text data, A, B, and C. The data volume of each group was 10. The number of single characters in the data of group A was less than 6; the number of single characters in the data of group B was between 6 and 15; and the number of single characters in the data of group C was greater than or equal to 15.

[0115] Build a traditional FastSpeech2 model and use a vocoder to play the speech data, and use the above text data for experiments. The generated data is used as the control group data.

[0116] The deep neural network model proposed in the embodiment of the present disclosure is constructed, and a vocoder is used to play the voice data, and the above text data is used for experiments. The generated data is used as the data of the experimental group.

[0117] The comparison results of the waiting time from inputting the text to be synthesized to playing the speech are as follows:

[0118] Group Control group Experimental Group A 67ms 39ms B 162ms 40ms C 241ms 39ms

[0119] The comparison results of the quality of the played voice data are as follows:

[0120] Group Control group Experimental Group A 4.20±0.05 4.17±0.03 B 4.08±0.07 4.02±0.06 C 3.97±0.05 3.88±0.07

[0121] Through the analysis of the above tables, using the deep neural network model proposed in the embodiment of the present disclosure and using a vocoder to play the voice data, as the amount of text increases, the waiting time from inputting the text to be synthesized to playing the voice changes little, and the quality of the played voice data is not much different from that of the traditional FastSpeech2 model and using a vocoder to play the voice data. That is, the voice playback method of the embodiment of the present disclosure can shorten the waiting time for voice playback while ensuring the quality of voice playback, thereby improving the efficiency of human-computer communication, even when the text to be synthesized is long.

[0122] See also Figure 4An exemplary embodiment of the present disclosure further provides a voice playback device 300, which is applied to an electronic device. The device 300 includes a coding module 310, a prediction module 320, a grouping module 330, a decoding module 340 and a playback module 350:

[0123] An encoding module 310 is used to encode each phoneme vector in the phoneme vector sequence corresponding to the text to be synthesized to obtain an encoding vector sequence;

[0124] A prediction module 320 is used to perform speech information prediction processing on the coding vector sequence to obtain speech information corresponding to each coding vector in the coding vector sequence;

[0125] A grouping module 330, configured to determine each decoding group and a decoding order corresponding to each decoding group based on the order of arrangement of each encoding vector in the encoding vector sequence and the decoding efficiency, wherein the decoding group includes at least one encoding vector and its corresponding speech information, and the decoding efficiency represents the number of encoding vectors to be decoded each time;

[0126] A decoding module 340, configured to decode each decoding packet based on a decoding order;

[0127] The playing module 350 is used for starting to play the decoded voice data in the order in which the voice data is decoded in response to the first voice data being decoded.

[0128] In some embodiments, the apparatus 300 further comprises:

[0129] The decoding efficiency determination module is used to determine the decoding efficiency based on a decoding efficiency configuration parameter, where the decoding efficiency configuration parameter is determined based on an acceptance level of the voice playback delay.

[0130] In some embodiments, the apparatus 300 further comprises:

[0131] The phoneme extraction module is used to extract phonemes from the text to be synthesized and obtain the phoneme sequence corresponding to the text to be synthesized.

[0132] The phoneme embedding module is used to perform phoneme embedding processing on the phoneme sequence to obtain a phoneme vector sequence.

[0133] In some implementations, the voice information includes a combination of one or more of duration, pitch, and volume.

[0134] In some embodiments, the encoding module is further used to encode each phoneme vector in the phoneme vector sequence through an encoder; the prediction module is further used to predict the speech information of the encoded vector sequence through a speech information predictor; the decoding module is further used to decode each decoding group in sequence through a decoder based on the decoding order. The encoder, speech information predictor and decoder are obtained through joint training based on sample speech data carrying text annotations.

[0135] In some embodiments, the apparatus 300 further comprises:

[0136] The first acquisition module is used to acquire interactive speech in a target environment.

[0137] The first conversion module is used to convert the interactive speech into a corresponding first interactive text.

[0138] The second acquisition module is used to acquire a first reply text corresponding to the first interaction text, and the first reply text is used as the text to be synthesized.

[0139] The third acquisition module is used to acquire the second interactive text input by the user.

[0140] The fourth acquisition module is used to acquire a second reply text corresponding to the second interactive text, and the second reply text is used as the text to be synthesized.

[0141] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0142] Figure 5 FIG. 4 is a block diagram of an electronic device 400 according to an exemplary embodiment. Figure 5 As shown, the electronic device 400 may include: a processor 401 and a memory 402. The electronic device 400 may also include one or more of a multimedia component 403, an input / output (I / O) interface 404, and a communication component 405.

[0143] The processor 401 is used to control the overall operation of the electronic device 400 to complete all or part of the steps in the above-mentioned voice playback method. The memory 402 is used to store various types of data to support the operation of the electronic device 400, and these data may include, for example, instructions for any application or method used to operate on the electronic device 400, and application-related data, such as contact data, sent and received messages, pictures, audio, video, etc. The memory 402 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 403 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 402 or sent through the communication component 405. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 404 provides an interface between the processor 401 and other interface modules, and the other interface modules may be keyboards, mice, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 405 is used for wired or wireless communication between the electronic device 400 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 405 may include: Wi-Fi module, Bluetooth module, NFC module, etc.

[0144] In an exemplary embodiment, the electronic device 400 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned voice playback method.

[0145] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by the processor, the steps of the above-mentioned voice playback method are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 402 including program instructions, and the above-mentioned program instructions can be executed by the processor 401 of the electronic device 400 to complete the above-mentioned voice playback method.

[0146] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a programmable device. The computer program has a code portion for executing the above-mentioned voice playback method when executed by the programmable device.

[0147] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0148] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0149] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A voice playing method, characterized in that: include: Encoding each phoneme vector in the phoneme vector sequence corresponding to the text to be synthesized to obtain an encoding vector sequence; Performing prediction processing on the coding vector sequence for voice information to obtain voice information corresponding to each coding vector in the coding vector sequence, wherein the voice information includes a combination of one or more of duration, pitch, and volume; Determining a decoding efficiency based on a decoding efficiency configuration parameter, wherein the decoding efficiency configuration parameter is determined based on an acceptance level of a voice playback delay; Determine each decoding group and a decoding order corresponding to each decoding group based on the order in which each encoding vector in the encoding vector sequence is arranged and the decoding efficiency, wherein the decoding group includes at least one encoding vector and its corresponding voice information, and the decoding efficiency represents the number of encoding vectors to be decoded each time; Based on the decoding order, decoding each of the decoding packets; In response to first voice data being obtained through decoding, starting to play the decoded voice data in the order in which the voice data is obtained through decoding; The decoding of each decoding group based on the decoding order includes: Decode one decoding packet each time, and after decoding the previous decoding packet, decode the next decoding packet; In response to obtaining the first voice data through decoding, starting to play the decoded voice data in the order in which the voice data is obtained through decoding, comprises: After decoding the decoding group located in the first decoding order to obtain the first voice data, the decoded voice data is played in the order in which the voice data is decoded.

2. The voice playing method according to claim 1, characterized in that: The method further comprises: Performing phoneme extraction on the text to be synthesized to obtain a phoneme sequence corresponding to the text to be synthesized; Perform phoneme embedding processing on the phoneme sequence to obtain the phoneme vector sequence.

3. The voice playing method according to claim 1, characterized in that: The encoding process of each phoneme vector in the phoneme vector sequence includes: Encoding each phoneme vector in the phoneme vector sequence by an encoder; The performing prediction processing of speech information on the coding vector sequence comprises: Performing speech information prediction processing on the coding vector sequence by a speech information predictor; The decoding of each decoding packet in sequence based on the decoding order includes: Based on the decoding order, decoding the respective decoding packets in sequence by a decoder; The encoder, speech information predictor and decoder are obtained through joint training based on sample speech data carrying text annotations.

4. The voice playing method according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire interactive speech in the target environment; Converting the interactive speech into a corresponding first interactive text; Acquire a first reply text corresponding to the first interactive text, and use the first reply text as the text to be synthesized; or Obtaining a second interactive text input by the user; A second reply text corresponding to the second interactive text is obtained, and the second reply text serves as the text to be synthesized.

5. A voice playing device, characterized in that: include: An encoding module, used for encoding each phoneme vector in the phoneme vector sequence corresponding to the text to be synthesized to obtain an encoding vector sequence; A prediction module, configured to perform prediction processing on the coding vector sequence for speech information to obtain speech information corresponding to each coding vector in the coding vector sequence, wherein the speech information includes a combination of one or more of duration, pitch and volume; A decoding efficiency determination module, configured to determine a decoding efficiency based on a decoding efficiency configuration parameter, wherein the decoding efficiency configuration parameter is determined based on an acceptance level of a voice playback delay; A grouping module, configured to determine each decoding group and a decoding order corresponding to each decoding group based on the order in which each encoding vector in the encoding vector sequence is arranged and the decoding efficiency, wherein the decoding group includes at least one encoding vector and its corresponding speech information, and the decoding efficiency represents the number of encoding vectors to be decoded each time; A decoding module, used for decoding each decoding packet based on the decoding order, and for decoding one decoding packet at a time, and decoding a subsequent decoding packet after decoding the previous decoding packet; A playback module is used to start playing the decoded voice data in response to the first voice data obtained by decoding in the order in which the voice data is decoded, and is used to start playing the decoded voice data in the order in which the voice data is decoded after decoding the decoding group located in the first decoding order to obtain the first voice data.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 4 are implemented.

7. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 4.

Images