Block-masked streaming speech conversion method

Abstract

The application relates to a block masking-based streaming speech conversion method. The method comprises the following steps: dividing a speech to be converted into at least one speech segment according to a preset window length, and determining block masking information of the speech to be converted according to the speech segment; determining speech content information corresponding to the speech to be converted according to the block masking information and the speech to be converted through a streaming encoder; the speech content information is irrelevant to speech object identity information corresponding to the speech to be converted; and the speech content information is decoded according to target speech object identity information through a streaming decoder to obtain converted speech corresponding to the speech to be converted. The method can be based on block masking and a streaming encoder and decoder, the influence of information related to a speaker on an encoding process can be reduced in a masking manner, information related to the speaker is avoided to be adopted, and therefore the accuracy of speech conversion is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a streaming voice conversion method and device based on block masking, a computer device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] Voice conversion (VC) is a technical means to change the identity of a speaker without changing the language content, and is widely used in voice coding and voice output devices. Voice conversion can convert the voice style (such as speaker identity, prosody, emotion, etc.) of a source voice to a target style while ensuring that the language content remains unchanged, thereby achieving conversion of the speaker identity.

[0003] In the conventional technology, an offline-trained voice conversion model is used for online voice conversion. However, the offline training and online inference scheme used in the conventional technology has the problem of asynchronization between training and inference. The features or types of the data in the offline training are inconsistent with those in the actual inference process. The model trained offline may treat information related to the speaker as part of the voice content, resulting in poor voice conversion results and adversely affecting the accuracy of voice conversion. SUMMARY

[0004] Therefore, it is necessary to provide a streaming voice conversion method and device based on block masking that can improve the accuracy of voice conversion.

[0005] In a first aspect, the present application provides a streaming voice conversion method based on block masking, comprising:

[0006] dividing a voice to be converted into at least one voice segment according to a preset window length, and determining block masking information of the voice to be converted according to the voice segment;

[0007] determining voice content information corresponding to the voice to be converted by a streaming encoder according to the block masking information and the voice to be converted; the voice content information is irrelevant to voice object identity information corresponding to the voice to be converted;

[0008] decoding the voice content information according to target voice object identity information by a streaming decoder to obtain converted voice corresponding to the voice to be converted.

[0009] In one embodiment, the block masking information of the voice to be converted is determined according to the voice segment, comprising:

[0010] screening a to-be-masked segment from the speech segment according to the preset window length, and determining block-level mask information according to the to-be-masked segment;

[0011] taking speech segments other than the to-be-masked segment in the speech segment as dynamic masking segments, and determining dynamic distribution mask information according to the dynamic masking segments;

[0012] determining the block masking information according to the block-level mask information and the dynamic distribution mask information.

[0013] In one of the embodiments, the determining, by the stream encoder, of the speech content information corresponding to the to-be-converted speech according to the block masking information and the to-be-converted speech comprises:

[0014] determining, by the stream encoder, the masking information corresponding to the to-be-converted speech according to the length masking information of the stream encoder and the block masking information;

[0015] determining, by the stream encoder, attention weight information according to the attention weight matrix corresponding to the stream encoder and the masking information;

[0016] determining the speech content information corresponding to the to-be-converted speech according to the attention weight information and the to-be-converted speech.

[0017] In one of the embodiments, the method further comprises:

[0018] acquiring, by the stream encoder, speech content information corresponding to a first speech segment in the to-be-converted speech;

[0019] taking the speech content information corresponding to the first speech segment as cache data, and splicing the cache data with a second speech segment in the to-be-converted speech to obtain a splicing result; the position of the second speech segment in the to-be-converted speech is earlier than that of the first speech segment;

[0020] determining, by the stream encoder, speech content information corresponding to the second speech segment according to the second speech segment and the splicing result.

[0021] In one of the embodiments, the decoding, by the stream decoder, of the speech content information according to the target speech object identity information to obtain the converted speech corresponding to the to-be-converted speech comprises:

[0022] decoding, by the stream decoder, the speech content information according to the target speech object identity information to obtain a decoding result corresponding to the to-be-converted speech;

[0023] acquiring a speech object identity classification result corresponding to the decoding result.

[0024] In a case where the voice object identity classification result matches the target voice object identity information, the decoding result is converted into converted voice corresponding to the voice to be converted.

[0025] In one of the embodiments, the method further comprises:

[0026] obtaining a first sample voice and a second sample voice; the first sample voice and the second sample voice have the same voice content; voice object identity information of the first sample voice and voice object identity information of the second sample voice are different;

[0027] training the stream encoder and the stream decoder according to the first sample voice and the second sample voice.

[0028] In a second aspect, the present application further provides a stream voice conversion device based on block masking, comprising:

[0029] an information obtaining module, configured to divide a voice to be converted according to a preset window length to obtain at least one voice segment, and determine block masking information of the voice to be converted according to the voice segment;

[0030] a content analyzing module, configured to determine voice content information corresponding to the voice to be converted by a stream encoder according to the block masking information and the voice to be converted; the voice content information is irrelevant to voice object identity information corresponding to the voice to be converted;

[0031] a voice decoding module, configured to decode the voice content information according to target voice object identity information by a stream decoder to obtain converted voice corresponding to the voice to be converted.

[0032] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to realize the steps of the above method.

[0033] In a fourth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the steps of the above method.

[0034] In a fifth aspect, the present application further provides a computer program product. The computer program product comprises a computer program, and the computer program is executed by the processor to realize the steps of the above method.

[0035] The block masking-based streaming speech conversion method, device, computer device, computer readable storage medium and computer program product can obtain at least one speech segment by dividing the speech to be converted according to a preset window length, determine the block masking information of the speech to be converted according to the speech segment, and gradually analyze the speech to be converted according to the preset window length to obtain accurate block masking information. The speech content information corresponding to the speech to be converted is determined by the streaming encoder according to the block masking information and the speech to be converted, so that the characteristic information irrelevant to the speaker in the speech to be converted is analyzed on the basis of the block masking information to obtain the speech content information irrelevant to the speech object identity information corresponding to the speech to be converted. The speech content information is decoded according to the target speech object identity information by the streaming decoder to obtain the converted speech corresponding to the speech to be converted, so that the speech content information is decoded in combination with the target speech object identity information to obtain the converted speech with the target speech object identity information as the speaker. The block masking and streaming encoder and decoder are used to reduce the influence of the information related to the speaker on the encoding process in a masking manner, avoid the information related to the speaker being adopted, and improve the accuracy of speech conversion. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other related drawings can be obtained by those skilled in the art without creative labor.

[0037] Figure 1 An application environment diagram of a block masking-based streaming speech conversion method in an embodiment;

[0038] Figure 2 A flowchart of a block masking-based streaming speech conversion method in an embodiment;

[0039] Figure 3 A flowchart of determining masking information in an embodiment;

[0040] Figure 4 A flowchart of streaming inference based on a streaming encoder in an embodiment;

[0041] Figure 5 A flowchart of generating converted speech in an embodiment;

[0042] Figure 6 A structural block diagram of a block masking-based streaming speech conversion device in an embodiment;

[0043] Figure 7This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] The block-masking-based streaming speech conversion method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on the cloud or other network servers. Server 104 divides the speech to be converted according to a preset window length, obtaining at least one speech segment, and determines the block masking information of the speech to be converted based on the speech segment. Server 104 uses a streaming encoder to determine the speech content information corresponding to the speech to be converted based on the block masking information and the speech to be converted. The speech content information is independent of the identity information of the speech object corresponding to the speech to be converted. Server 104 uses a streaming decoder to decode the speech content information based on the identity information of the target speech object, obtaining the converted speech corresponding to the speech to be converted. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Headset devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0046] In one exemplary embodiment, such as Figure 2 As shown, a block-masking-based streaming speech conversion method is provided. Taking the application of this method to a server as an example, the method includes the following steps S202 to S206. Wherein:

[0047] Step S202: Divide the speech to be converted into segments according to the preset window length to obtain at least one speech segment, and determine the block masking information of the speech to be converted based on the speech segments.

[0048] The preset window length can be a length of a window function pre-set for analyzing the voice signal of the voice to be converted. In actual application, the preset window length can be represented as W.

[0049] The voice to be converted can be a voice that needs to be converted. In actual application, the voice to be converted can be a voice containing human voice.

[0050] The voice segment can be a voice obtained by dividing the voice to be converted according to the preset window length.

[0051] The block masking information can be information representing the influence degree of each voice segment on voice conversion. In actual application, the block masking information can be used to identify or distinguish irrelevant information in the voice to be converted. The irrelevant information can include speaker identity information (i.e., voice object identity information).

[0052] As an example, when the voice to be converted needs to be converted, the server can input the voice to be converted into a pre-trained block masking based streaming voice conversion acoustic model. The block masking based streaming voice conversion acoustic model can use a block masking based streaming voice conversion method. The voice to be converted is first divided according to the preset window length to obtain at least one voice segment. Then, the server analyzes the voice segment by using the block masking based streaming voice conversion acoustic model, and determines the block masking information of the voice to be converted.

[0053] In step S204, the voice content information corresponding to the voice to be converted is determined by the streaming encoder according to the block masking information and the voice to be converted. The voice content information is irrelevant to the voice object identity information corresponding to the voice to be converted.

[0054] The streaming encoder can be a model used for encoding / modeling the voice to be converted.

[0055] The voice content information can be information representing the voice content of the voice to be converted. In actual application, the voice content information can include information describing the voice content. In specific implementation, the voice content information can be irrelevant to the voice object identity information corresponding to the voice to be converted.

[0056] The voice object identity information corresponding to the voice to be converted can be information representing the identity of the speaker corresponding to the voice to be converted. For example, an audio of a character A reading a certain text is recorded as the voice to be converted, and the character A can be the speaker of the voice to be converted.

[0057] As an example, the server utilizes a streaming encoder in a streaming speech conversion acoustic model based on block masking to analyze and model the to-be-converted speech in combination with the block masking information, and determines the speech content information corresponding to the to-be-converted speech. Since the streaming encoder is pre-trained, it can be considered that the speech content information output by the streaming encoder is irrelevant to the speech object identity information corresponding to the to-be-converted speech. In a specific implementation, the streaming encoder can use an attention mechanism to determine the information related to the speaker in the to-be-converted speech as irrelevant information, ignore the influence of the irrelevant information on the encoding / modeling result in the encoding / modeling process, and focus on the feature information other than the irrelevant information in the to-be-converted speech, thereby obtaining the speech content information irrelevant to the speech object identity information corresponding to the to-be-converted speech.

[0058] In step S206, the speech content information is decoded by a streaming decoder according to the target speech object identity information to obtain the converted speech corresponding to the to-be-converted speech.

[0059] The streaming decoder can be a model for decoding / converting the speech content information into speech.

[0060] The target speech object identity information can be information representing the identity of the speaker of the converted speech. In actual applications, the speech conversion can convert the speaker of the to-be-converted speech from the original speaker A0 to the target speaker A1. The identity information of the original speaker A0 can be used as the speech object identity information of the to-be-converted speech, and the identity information of the target speaker A1 can be used as the target speech object identity information. In a specific implementation, the target speech object identity information can be specified by a user.

[0061] The converted speech can be the speech obtained after the speech conversion of the to-be-converted speech. In actual applications, the speech object identity information of the converted speech can include the target speech object identity information.

[0062] As an example, the server can utilize a streaming decoder in a streaming speech conversion acoustic model based on block masking to obtain the target speech object identity information of the speech conversion character corresponding to the to-be-converted speech. Then, the server utilizes the streaming decoder to decode the speech content information according to the target speech object identity information to obtain the converted speech corresponding to the to-be-converted speech. In actual applications, the streaming decoder can refer to a discriminator to obtain the accuracy of the speaker identity corresponding to the converted speech. In the decoding process of the streaming decoder, when the discriminator determines that the speaker identity corresponding to the converted speech is accurate enough, the server regards the converted speech at this time as the final output speech.

[0063] In the speech conversion method based on the block masking, the speech to be converted is divided according to the preset window length to obtain at least one speech segment, and the block masking information of the speech to be converted is determined according to the speech segment, so that the speech to be converted is analyzed step by step according to the preset window length to obtain accurate block masking information; the speech content information corresponding to the speech to be converted is determined by the streaming encoder according to the block masking information and the speech to be converted, so that the characteristic information irrelevant to the speaker in the speech to be converted is analyzed on the basis of the block masking information to obtain the speech content information irrelevant to the speech object identity information corresponding to the speech to be converted; the speech content information is decoded according to the target speech object identity information by the streaming decoder to obtain the converted speech corresponding to the speech to be converted, so that the speech content information is decoded in combination with the target speech object identity information to obtain the converted speech with the target speech object identity information as the speaker, which can reduce the influence of the information related to the speaker on the encoding process in a masking manner based on the block masking and the streaming encoder and decoder, avoid the information related to the speaker being adopted, and thus improve the accuracy of speech conversion.

[0064] In an exemplary embodiment, determining the block masking information of the speech to be converted according to the speech segment includes: screening a to-be-masked segment from the speech segment according to the preset window length, and determining block-level mask information according to the to-be-masked segment; taking the speech segment other than the to-be-masked segment as a dynamic masking segment, and determining dynamic distribution mask information according to the dynamic masking segment; and determining the block masking information according to the block-level mask information and the dynamic distribution mask information.

[0065] In the method, the to-be-masked segment can be any speech segment with the preset window length in the speech segments obtained by dividing the speech to be converted according to the preset window length.

[0066] The block-level mask information can be information for controlling the influence degree of the speech segment located after any segment (such as a current speech segment) in the speech segments corresponding to the speech to be converted on the speech content information.

[0067] The dynamic masking segment can be a speech segment other than the to-be-masked segment in the speech to be converted.

[0068] The dynamic distribution mask information can be information for controlling the influence degree of the speech segment other than the to-be-masked segment in the speech to be converted on the speech content information.

[0069] As an example, the server can utilize the block masking-based streaming speech conversion acoustic model to divide the speech to be converted into a plurality of speech segments according to a preset window length, the speech segments with a length of the preset window length in the speech segments can be taken as the to-be-masked segments, the speech segments other than the to-be-masked segments in the speech segments can be taken as the dynamic masking segments, the server can analyze the to-be-masked segments one by one as the current research object, specifically, the server can utilize the block masking-based streaming speech conversion acoustic model to determine the block-level mask information mask chunk for the to-be-masked segment currently in need of analysis. chunk The speech segments after the to-be-masked segment currently in need of analysis can be masked, the server can determine the dynamic distribution mask information mask seq for the dynamic masking segment. seq The dynamic masking segment can be masked, the server can determine the block masking information mask chunk according to the block-level mask information mask seq and the dynamic distribution mask information mask dynamic , in actual application, the calculation expression of the block masking information mask dynamic may be expressed as:

[0070] .

[0071] In the embodiment, by screening the to-be-masked segments from the speech segments according to the preset window length, determining the block-level mask information according to the to-be-masked segments, taking the speech segments other than the to-be-masked segments in the speech segments as the dynamic masking segments, determining the dynamic distribution mask information according to the dynamic masking segments, and determining the block masking information according to the block-level mask information and the dynamic distribution mask information, each speech segment in the speech to be converted can be analyzed one by one, accurate block-level mask information and dynamic distribution mask information can be determined, and thus accurate block masking information can be obtained, which provides a data basis for subsequent determination of speech content information of the speech to be converted by the streaming encoder using the block masking information, and thus the accuracy of speech conversion is improved.

[0072] In some embodiments, the streaming encoder determines the speech content information corresponding to the speech to be converted according to the block masking information and the speech to be converted, including: the streaming encoder determines the masking information corresponding to the speech to be converted according to the length masking information and the block masking information of the streaming encoder; the streaming encoder determines the attention weight information according to the attention weight matrix corresponding to the streaming encoder and the masking information; and the streaming encoder determines the speech content information corresponding to the speech to be converted according to the attention weight information and the speech to be converted.

[0073] The length masking information can be information for controlling the influence degree of the length of the to-be-masked segment on the speech content information.

[0074] The masking information can be information for controlling the influence degree of any information / any data (such as a speech segment) in the to-be-converted speech on the speech content information.

[0075] The attention weight matrix can be information representing the importance between any information / any data in the to-be-converted speech.

[0076] The attention weight information can be information representing the importance of any information / any data (such as a speech segment) in the to-be-converted speech on the speech content information.

[0077] As an example, the server can determine the masking information mask length and the block masking information mask dynamic of the to-be-converted speech through the stream encoder according to the length masking information mask final of the stream encoder. In actual application, the calculation expression of the masking information mask final may be:

[0078] .

[0079] Then the server can determine the attention weight information through the stream encoder according to the attention weight matrix and the masking information mask final of the stream encoder, and determine the speech content information corresponding to the to-be-converted speech according to the attention weight information and the to-be-converted speech.

[0080] In this embodiment, the masking information of the to-be-converted speech is determined through the stream encoder according to the length masking information and the block masking information of the stream encoder; the attention weight information is determined through the stream encoder according to the attention weight matrix and the masking information of the stream encoder; and the speech content information corresponding to the to-be-converted speech is determined according to the attention weight information and the to-be-converted speech. Therefore, the accurate masking information can be obtained based on the length masking information and the block masking information of the stream encoder, and the speech content information can be accurately generated by combining the attention weight matrix and the attention weight information, so as to improve the accuracy of the speech content information, provide a data basis for subsequent decoding of the speech content information to realize speech conversion, and thus improve the accuracy of the speech conversion.

[0081] In some embodiments, the method further comprises: obtaining, by the streaming encoder, speech content information corresponding to the first speech segment in the speech to be converted; taking the speech content information corresponding to the first speech segment as cache data, and splicing the cache data with a second speech segment in the speech to be converted to obtain a splicing result; the position of the second speech segment in the speech to be converted is in front of the first speech segment; and determining, by the streaming encoder, speech content information corresponding to the second speech segment according to the second speech segment and the splicing result.

[0082] The first speech segment can be any speech segment with a preset window length among a plurality of speech segments obtained by dividing the speech to be converted according to the preset window length.

[0083] The splicing result can be data obtained after splicing the cache data with the second speech segment in the speech to be converted.

[0084] The second speech segment can be a speech segment located after the first speech segment in the speech to be converted. In actual applications, the position relationship / arrangement order among a plurality of speech segments obtained by dividing the speech to be converted according to a preset window length corresponds to the position of the speech segment in the speech to be converted. For example, if the speech to be converted with a total duration of 25 seconds is divided every 10 seconds, three speech segments are obtained, the first speech segment corresponds to the content from the 1st second to the 10th second in the speech to be converted, the second speech segment corresponds to the content from the 11th second to the 20th second in the speech to be converted, and the third speech segment corresponds to the content from the 21st second to the 25th second in the speech to be converted. The first speech segment can be taken as the first speech segment, and the second speech segment can be taken as the second speech segment.

[0085] As an example, the server can employ a cache mechanism, obtain speech content information corresponding to the first speech segment in the speech to be converted by using the streaming encoder, take the speech content information corresponding to the first speech segment as cache data, then splice the cache data with a second speech segment in the speech to be converted to obtain a splicing result, and take the splicing result as the key vector K and the value vector V. The streaming encoder determines speech content information corresponding to the second speech segment according to the second speech segment and the splicing result.

[0086] In this embodiment, the speech content information corresponding to the first speech segment in the to-be-converted speech is obtained through the streaming encoder; the speech content information corresponding to the first speech segment is taken as the cache data, and the cache data is spliced with the second speech segment in the to-be-converted speech to obtain a splicing result; the position of the second speech segment in the to-be-converted speech is in front of the first speech segment; the speech content information corresponding to the second speech segment is determined through the streaming encoder according to the second speech segment and the splicing result, the cache mechanism can be used to splice the speech content information corresponding to the first speech segment with the second speech segment in the to-be-converted speech, the causal coherence of the inference process is ensured, accurate speech content information is obtained, and the accuracy of speech conversion is improved.

[0087] In some embodiments, the to-be-converted speech corresponding to the converted speech is obtained by decoding the speech content information according to the target speech object identity information through the streaming decoder, including: decoding the speech content information according to the target speech object identity information through the streaming decoder to obtain a decoding result corresponding to the to-be-converted speech; obtaining a speech object identity classification result corresponding to the decoding result; in the case where the speech object identity classification result matches the target speech object identity information, converting the decoding result into the to-be-converted speech corresponding to the converted speech.

[0088] In the actual application, the decoding result can include audio features such as mel-frequency spectrum.

[0089] In the actual application, the speech object identity classification result can be generated by a speaker classifier / discriminator based on the decoding result, and the speech object identity classification result can reflect / describe the authenticity and conversion effect of speech conversion.

[0090] As an example, the server can decode the voice content information according to the target voice object identity information through the streaming decoder to obtain a decoding result corresponding to the voice to be converted, then the server can obtain a voice object identity classification result corresponding to the decoding result by using a preset speaker classifier / discriminator, and the server can compare the voice object identity classification result with the target voice object identity information. In the case where the voice object identity classification result matches the target voice object identity information, the server can determine that the speaker identity feature of the decoding result at this time is sufficient to be real, and the server can convert the decoding result into converted voice corresponding to the voice to be converted. In the case where the voice object identity classification result does not match the target voice object identity information, the server can determine that the speaker identity feature of the decoding result at this time does not meet the corresponding voice conversion requirement, and the server can divide the voice to be converted according to a new window length by using a block masking-based streaming voice conversion acoustic model, and obtain a new decoding result by using a streaming encoder and a streaming decoder until the speaker identity feature of the new decoding result is sufficient to be real.

[0091] In this embodiment, the voice content information is decoded according to the target voice object identity information through the streaming decoder to obtain a decoding result corresponding to the voice to be converted, a voice object identity classification result corresponding to the decoding result is obtained, and in the case where the voice object identity classification result matches the target voice object identity information, the decoding result is converted into converted voice corresponding to the voice to be converted. The voice object identity classification result corresponding to the decoding result can be analyzed in time to match the target voice object identity information, the converted voice is ensured to have accurate target voice object identity information, and thus the accuracy of voice conversion is improved.

[0092] In some embodiments, the above method further includes: obtaining a first sample voice and a second sample voice; the first sample voice and the second sample voice have the same voice content; the voice object identity information of the first sample voice and the voice object identity information of the second sample voice are different; and the streaming encoder and the streaming decoder are trained according to the first sample voice and the second sample voice.

[0093] As an example, in order to optimize the block mask-based streaming speech conversion acoustic model to achieve accurate speech conversion, the server can train the block mask-based streaming speech conversion acoustic model using training samples. Specifically, the server can obtain a first sample speech and a second sample speech, wherein the first sample speech and the second sample speech have the same speech content, and the speech object identity information of the first sample speech and the speech object identity information of the second sample speech are different. Then, the server can train the streaming encoder and the streaming decoder in the block mask-based streaming speech conversion acoustic model according to the first sample speech and the second sample speech until the model loss function value corresponding to the block mask-based streaming speech conversion acoustic model meets the preset loss function requirement, and obtain a pre-trained block mask-based streaming speech conversion acoustic model.

[0094] In the embodiment, by obtaining the first sample speech and the second sample speech, the first sample speech and the second sample speech have the same speech content, the speech object identity information of the first sample speech and the speech object identity information of the second sample speech are different, and the streaming encoder and the streaming decoder are trained according to the first sample speech and the second sample speech. The block mask-based streaming speech conversion acoustic model can be trained using training samples, so as to ensure the model performance of the block mask-based streaming speech conversion acoustic model, and further improve the accuracy of speech conversion.

[0095] In some embodiments, the server can implement speech conversion by means of the block mask-based streaming speech conversion acoustic model. The block mask-based streaming speech conversion acoustic model can extract speaker-independent feature information by combining offline and online training of the dynamic block mask streaming Transformer speech recognition model as a streaming encoder, and extract mel features by the feedforward Transformer structure in FastSpeech as a streaming decoder. In addition, the block mask-based streaming speech conversion acoustic model can further include a speaker adversarial network (such as a speaker classifier / discriminator) to further remove residual speaker information.

[0096] Firstly, the server can randomly sample the window size W from the uniform distribution U(1, max_length), and divide the speech to be converted according to the window size W. The speech segment obtained by the block division can be regarded as a block, and the tail part of the speech to be converted which cannot be evenly divided by W will be dynamically distributed with mask information mask seq Masking, block-level mask information mask chunkThe current block can be masked, so that only the attention weight between the current block and the previous block is calculated, and the current block is not affected by the future input information. Since the attention calculated within the block is non-causal, the partial output can see part of the future input information. In this way, the quality of voice conversion can be improved. In actual application, the server can use the cross attention mechanism when obtaining the masking information. At the same time, the server can also flexibly adjust the size of the window length W (such as the inference block) according to specific needs. The length masking information mask length It can be used in batch training and inference applications of neural networks. The length of each batch of data is masked, as shown in Figure 3 A flowchart for determining block masking information is provided. The server can determine the block masking information mask chunk and dynamic distribution mask information mask seq , determine the masking information mask dynamic of the to-be-converted voice in combination with the length masking information mask length of the streaming encoder. final , Figure 3 In the above process, the white blocks represent the masked part. In the calculation of the neural network, the tensor data is set to 0. It can be understood that the training process of the streaming voice conversion acoustic model based on block masking is similar to the above inference application process.

[0097] As an example, the streaming encoder can be stacked by multiple Transformer encoder layers, and the online and offline systems share the same encoder. Dynamic block masking is applied in the shared encoder. The training loss of the streaming encoder can include the joint loss L combine (x, y) of CTC and attention encoder-decoder. combine The calculation expression of the joint loss L combine (x, y) can be expressed as:

[0098] .

[0099] Where x can represent acoustic features, y can be the text annotation corresponding to the acoustic features, L CTC (x, y) can refer to the ctc loss, L AED (x, y) can refer to the loss of the attention encoder-decoder, and λ can be a preset hyperparameter for balancing the loss of the CTC and attention mechanism.

[0100] The streaming encoder can model the input audio (such as the to-be-converted voice) into intermediate features (such as voice content information), and then the streaming decoder generates a mel spectrum based on the intermediate features, and further generates the converted voice based on the mel spectrum, as shown in Figure 4As shown, a flowchart of a streaming inference based on a streaming encoder is provided. In order to ensure that the streaming speech conversion acoustic model based on block masking is causal and to speed up the inference speed, the server can use a caching mechanism in the streaming encoder. In the encoder layer of the streaming encoder, for the output of the historical encoder layer at the last time, the streaming encoder takes the output of the historical encoder layer at the last time as the cache data, splices it with the input of the current encoder layer as the key vector K and the value vector V, and the query vector Q is still the input of the current encoder layer.

[0101] The streaming decoder can use the feedforward Transformer structure in FastSpeech. The feedforward layer is composed of two one-dimensional convolutional neural networks with ReLU activation function. In order to ensure the causality of streaming inference, the server can add dynamic block masking during the training of the streaming decoder. The one-dimensional convolution in the feedforward layer is replaced by one-dimensional causal convolution, as shown in the following formula: Figure 5 As shown, a flowchart of generating converted speech is provided. The streaming decoder can include several streaming decoder layers. The speaker adversarial network is added to the streaming speech conversion acoustic model based on block masking. The speaker adversarial network can include a speaker classifier and a gradient reversal layer. The speaker classifier is a simple convolutional structure. The speaker adversarial classification is performed on each frame input (such as the decoding result) in the time dimension. The gradient reversal layer is added between the output of each decoder layer and the speaker classification network. The speaker label (such as the target speech object identity information) is sent to the next decoder layer together with the output of each decoder layer.

[0102] When training the streaming speech conversion acoustic model based on block masking, the training loss formula of the streaming speech conversion acoustic model based on block masking can be represented as:

[0103] .

[0104] Where X real represents the real mel spectrum, X gen represents the generated mel spectrum, V out represents the speech object identity classification result output by the speaker classifier, V spk represents the real label (such as the target speech object identity information), L mel represents the mel spectrum loss, L spk represents the speaker loss, and L total represents the total training loss. When the total training loss L totalWhen the value of the loss function is less than a preset threshold, the server determines that the training of the block masking based streaming voice conversion acoustic model at this time is completed, and the server can use the block masking based streaming voice conversion acoustic model at this time as a pre-trained block masking based streaming voice conversion acoustic model.

[0105] In this embodiment, the block masking based streaming voice conversion model is improved from input delay and processing delay. By setting a dynamic masking method, using a pre-trained speech recognition joint model as an encoder and a dynamic block masking based offline voice conversion model as a decoder, online voice conversion of any speaker to multiple speakers within 200 ms delay can be realized, thereby improving the accuracy of voice conversion.

[0106] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0107] Based on the same inventive concept, the embodiments of the present application also provide a block masking based streaming voice conversion device for implementing the above-mentioned block masking based streaming voice conversion method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more block masking based streaming voice conversion device embodiments provided below can refer to the limitations of the block masking based streaming voice conversion method in the above text, which will not be repeated here.

[0108] In one exemplary embodiment, as shown in Figure 6 a block masking based streaming voice conversion device is provided, comprising: an information acquisition module 602, a content analysis module 604 and a speech decoding module 606, wherein:

[0109] The information acquisition module 602 is configured to divide the voice to be converted into at least one voice segment according to a preset window length, and determine the block masking information of the voice to be converted according to the voice segment.

[0110] The content analysis module 604 is configured to determine speech content information corresponding to the speech to be converted by the stream encoder according to the block masking information and the speech to be converted.

[0111] The speech decoding module 606 is configured to decode the speech content information according to the target speech object identity information by the stream decoder to obtain converted speech corresponding to the speech to be converted.

[0112] In one of the example embodiments, the information acquisition module 602 is further configured to filter out a to-be-masked segment from the speech segment according to the preset window length, determine block-level mask information according to the to-be-masked segment, and determine dynamic distribution mask information according to a dynamic masking segment that is a speech segment other than the to-be-masked segment in the speech segment, and determine the block masking information according to the block-level mask information and the dynamic distribution mask information.

[0113] In one of the example embodiments, the content analysis module 604 is further configured to determine masking information corresponding to the speech to be converted by the stream encoder according to length masking information of the stream encoder and the block masking information, determine attention weight information by the stream encoder according to the attention weight matrix corresponding to the stream encoder and the masking information, and determine the speech content information corresponding to the speech to be converted according to the attention weight information and the speech to be converted.

[0114] In one of the example embodiments, the content analysis module 604 is further configured to obtain speech content information corresponding to a first speech segment in the speech to be converted by the stream encoder, take the speech content information corresponding to the first speech segment as cache data, splice the cache data with a second speech segment in the speech to be converted to obtain a splicing result, the second speech segment is located in front of the first speech segment in the speech to be converted, and determine speech content information corresponding to the second speech segment by the stream encoder according to the second speech segment and the splicing result.

[0115] In one of the example embodiments, the speech decoding module 606 is further configured to decode the speech content information according to the target speech object identity information by the stream decoder to obtain a decoding result corresponding to the speech to be converted, obtain speech object identity classification results corresponding to the decoding result, and convert the decoding result into converted speech corresponding to the speech to be converted in a case where the speech object identity classification results match the target speech object identity information.

[0116] In one of the example embodiments, the device further comprises a training module, which is specifically configured to obtain a first sample voice and a second sample voice; the first sample voice and the second sample voice have the same voice content; the voice object identity information of the first sample voice and the voice object identity information of the second sample voice are different; and the stream encoder and the stream decoder are trained according to the first sample voice and the second sample voice.

[0117] The modules in the above block masking-based stream voice conversion device can be implemented by software, hardware, or a combination thereof, in whole or in part. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so as to be invoked and executed by the processor to perform the operations corresponding to the modules.

[0118] In one of the example embodiments, a computer device is provided, which can be a terminal, and an internal structure diagram of the computer device can be as shown in Figure 7 The computer device comprises a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to perform wired or wireless communication with external terminals, and the wireless communication can be achieved through WIFI, mobile cellular network, near field communication (NFC), or other technologies. The computer program is executed by the processor to implement a block masking-based stream voice conversion method. The display unit of the computer device is configured to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or can be a key, trackball, or touchpad arranged on the shell of the computer device, or can be an external keyboard, touchpad, or mouse, etc.

[0119] Those skilled in the art can understand that Figure 7The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0120] In an embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method embodiments when executing the computer program.

[0121] In an embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0122] In an embodiment, a computer program product is provided, including a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0123] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0125] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0126] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A streaming speech conversion method based on block masking, characterized in that, The method includes: The speech to be converted is input into a pre-trained block-masking-based streaming speech-to-speech acoustic model. The streaming speech-to-speech acoustic model divides the speech to be converted into at least one speech segment according to a preset window length, and determines the block masking information of the speech to be converted based on the speech segment. The streaming speech-to-speech acoustic model includes a streaming encoder and a streaming decoder. The streaming decoder includes several streaming decoder layers. The streaming speech-to-speech acoustic model includes a speaker adversarial network, which includes a speaker classifier and a gradient inversion layer. The speaker classifier includes a convolutional structure and performs speaker adversarial classification on each frame of input in the temporal dimension. The gradient inversion layer is located between the output of each streaming decoder layer and the speaker classifier. The speaker label is fed into the next streaming decoder layer along with the output of each streaming decoder layer. The streaming encoder determines the speech content information corresponding to the speech to be converted based on the block masking information and the speech to be converted. The speech content information is unrelated to the identity information of the speech object corresponding to the speech to be converted. The streaming encoder is used to determine the speaker-related information in the speech to be converted as irrelevant information based on the block masking information through an attention mechanism, and ignores the influence of the irrelevant information on the encoding result during the encoding process, so as to obtain speech content information unrelated to the identity information of the speech object corresponding to the speech to be converted. The streaming decoder decodes the speech content information based on the target speech object's identity information to obtain the converted speech corresponding to the speech to be converted.

2. The method according to claim 1, characterized in that, Determining the block masking information of the speech to be converted based on the speech segment includes: Based on the preset window length, the segments to be masked are selected from the speech segments, and block-level mask information is determined based on the segments to be masked; The speech segments other than the segment to be masked are taken as dynamic masking segments, and dynamic distribution mask information is determined based on the dynamic masking segments; The block masking information is determined based on the block-level masking information and the dynamic distribution masking information.

3. The method according to claim 1, characterized in that, The step of determining the speech content information corresponding to the speech to be converted by the streaming encoder based on the block masking information and the speech to be converted includes: The streaming encoder determines the masking information corresponding to the speech to be converted based on the length masking information and the block masking information of the streaming encoder. The attention weight information is determined by the streaming encoder based on the attention weight matrix corresponding to the streaming encoder and the masking information; Based on the attention weight information and the speech to be converted, the speech content information corresponding to the speech to be converted is determined.

4. The method according to claim 3, characterized in that, The method further includes: The streaming encoder obtains the audio content information corresponding to the first audio segment in the audio to be converted. The audio content information corresponding to the first audio segment is used as cached data, and the cached data is concatenated with the second audio segment in the audio to be converted to obtain a concatenation result; the second audio segment is positioned before the first audio segment in the audio to be converted. The streaming encoder determines the audio content information corresponding to the second audio segment based on the second audio segment and the splicing result.

5. The method according to claim 1, characterized in that, The step of decoding the speech content information according to the target speech object identity information using the streaming decoder to obtain the converted speech corresponding to the speech to be converted includes: The streaming decoder decodes the speech content information according to the identity information of the target speech object to obtain the decoding result corresponding to the speech to be converted; Obtain the voice object identity classification result corresponding to the decoding result; If the voice object identity classification result matches the target voice object identity information, the decoding result is converted into the converted voice corresponding to the voice to be converted.

6. The method according to claim 1, characterized in that, The method further includes: Obtain a first sample speech and a second sample speech; the first sample speech and the second sample speech have the same speech content; the speech object identity information of the first sample speech and the speech object identity information of the second sample speech are different; The streaming encoder and the streaming decoder are trained based on the first sample speech and the second sample speech.

7. A streaming speech conversion device based on block masking, characterized in that, The device includes: An information acquisition module is used to input the speech to be converted into a pre-trained block-masking-based streaming speech-to-speech acoustic model. The streaming speech-to-speech acoustic model divides the speech to be converted into at least one speech segment according to a preset window length, and determines the block masking information of the speech to be converted based on the speech segment. The streaming speech-to-speech acoustic model includes a streaming encoder and a streaming decoder. The streaming decoder includes several streaming decoder layers. The streaming speech-to-speech acoustic model includes a speaker adversarial network. The speaker adversarial network includes a speaker classifier and a gradient inversion layer. The speaker classifier includes a convolutional structure and performs speaker adversarial classification on each frame of input in the time dimension. The gradient inversion layer is located between the output of each streaming decoder layer and the speaker classifier. The speaker label is fed into the next streaming decoder layer along with the output of each streaming decoder layer. The content analysis module is used to determine the speech content information corresponding to the speech to be converted based on the block masking information and the speech to be converted by the streaming encoder; the speech content information is unrelated to the identity information of the speech object corresponding to the speech to be converted; the streaming encoder is used to determine the speaker-related information in the speech to be converted as irrelevant information based on the block masking information by an attention mechanism, and ignore the influence of the irrelevant information on the encoding result during the encoding process, so as to obtain speech content information unrelated to the identity information of the speech object corresponding to the speech to be converted; The speech decoding module is used to decode the speech content information according to the identity information of the target speech object through the streaming decoder to obtain the converted speech corresponding to the speech to be converted.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images