A video conversion method and system based on mask prediction

Abstract

This application provides a video conversion method and system based on mask prediction. The method involves extracting the audio to be converted from the video, obtaining speaker information corresponding to the audio text in a first language and the speaker's audio based on the audio, and then performing language conversion on the audio text in a second language different from the first language to obtain the text to be converted. A first audio is generated based on the text to be converted, and a second audio is retrieved based on the speaker information. The first and second audios are input into an acoustic model, which, trained using mask prediction, performs speech feature replacement based on the features of the first and second audios, outputting a target audio. The target audio replaces the audio to be converted, resulting in the target video. This application improves the accuracy of speech feature extraction by the acoustic model through mask prediction, thereby improving the accuracy of speech feature extraction from the first and second audios and enhancing the synthesis effect of the audio after speech translation.

Description

[0001] This application claims priority to Chinese patent application No. 202311479579.X, filed on November 8, 2023, with the State Intellectual Property Office of China. The entire contents of the latter are incorporated herein by reference. Technical Field

[0002] This application relates to the field of speech conversion technology, and in particular to a video conversion method and system based on mask prediction. Background Technology

[0003] When watching films from different countries, users cannot directly understand the meaning of the dialogue from the audio alone because the film's language differs from their own. Therefore, users need to continuously read subtitles while watching the film to comprehend the meaning of the characters' lines.

[0004] To improve the viewing experience, the dialogue can be translated into other languages ​​before being dubbed in post-production, allowing users to intuitively understand the meaning of the lines based on the dubbing. However, this post-dubbing method is time-consuming and needs to be completed before the video plays, making it impossible to translate and play the dialogue in real time.

[0005] Therefore, speech-to-text conversion technology can be used to convert the language of a film into the user's desired language while preserving the vocal characteristics of the characters, thus achieving real-time translation. To achieve this, the speech-to-text conversion model needs to be trained with a large number of voice samples from specific individuals. However, it is difficult to obtain a large number of voice samples for a particular individual, resulting in low accuracy of speech-to-text conversion in scenarios with few samples. Summary of the Invention

[0006] To address the issue of low speech conversion accuracy when translating spoken language in scenarios with low sample counts, some embodiments of this application provide a video conversion method and system based on mask prediction.

[0007] In a first aspect, some embodiments of this application provide a video conversion method based on mask prediction, including:

[0008] Extract the audio to be converted from the video to be converted, wherein the audio to be converted includes at least one speaker's audio;

[0009] Based on the audio to be converted, audio text and speaker information corresponding to the speaker's audio are obtained, wherein the language information of the audio text is the first language;

[0010] The audio text is converted to a second language, which is different from the first language, to obtain the text to be converted.

[0011] A first audio is generated based on the text to be converted, and a second audio is retrieved based on the speaker information;

[0012] The first audio and the second audio are input into an acoustic model to perform speech feature replacement based on the first audio feature and the second audio feature, and output the target audio. The first audio feature is the audio feature corresponding to the first audio, and the second audio feature is the audio feature corresponding to the second audio. The acoustic model is obtained through training by mask prediction. The mask prediction is used to represent masking at least part of the training data and to predict the masked training samples during the training process.

[0013] Replace the audio to be converted with the target audio to obtain the target video.

[0014] In some embodiments, the acoustic model includes an audio encoding module and an audio decoding module, the audio encoding module includes a feature encoding unit and a permutation unit, and after the step of inputting the first audio and the second audio into the acoustic model, the method further includes:

[0015] The first audio feature of the first audio is extracted by the feature encoding unit, and the second audio feature of the second audio is extracted by the feature encoding unit.

[0016] The permutation unit calculates a feature similarity value, which is the similarity value between the second audio feature and the first audio feature. Based on the feature similarity value, a third audio feature is extracted from the second audio feature, and the first audio feature is replaced by the third audio feature to obtain the target audio feature. The third audio feature is a second audio feature whose feature similarity value is greater than or equal to a similarity threshold.

[0017] The target audio is obtained by decoding the target audio features through the audio decoding module.

[0018] In some embodiments, the step of calculating feature similarity values ​​through the permutation unit includes:

[0019] The permutation unit generates a fourth audio feature based on the first audio feature, and the fourth audio feature is generated according to the non-nearest neighbor replacement algorithm.

[0020] According to the proximity algorithm, the proximity value between the second audio feature and the fourth audio feature is calculated;

[0021] Feature similarity values ​​are generated based on the neighbor values.

[0022] In some embodiments, prior to the step of inputting the first audio and the second audio into the acoustic model, the method further includes:

[0023] Obtain a first training dataset, which includes a preset number of first training data.

[0024] The first training data is input into the audio encoding module to be trained, so as to extract the first training feature of the first training data through the feature encoding unit.

[0025] At least two first training features are input into the permutation unit to perform speech feature replacement on the first training features to obtain second training features;

[0026] Calculate the feature loss between the second training feature and the audio feature label, where the audio feature label is used to characterize standard audio features;

[0027] If the feature loss is less than or equal to the first loss threshold, the audio encoding module is output according to the current parameters of the feature encoding unit and the current parameters of the permutation unit;

[0028] If the feature loss is greater than the first loss threshold, then the feature encoding unit and the permutation unit are iteratively trained based on the first training data.

[0029] In some embodiments, the feature encoding unit includes a mask subunit, a first encoding unit, and a second encoding unit. The step of extracting the first training feature from the first training data through the feature encoding unit includes:

[0030] The first training data is input into the feature encoding unit so that the first training data is masked by the masking subunit to obtain masked training data.

[0031] The first encoding unit performs feature extraction on the mask training data to obtain the mask audio features;

[0032] The second encoding unit performs feature prediction on the masked portion of the masked audio features to obtain the first training features.

[0033] In some embodiments, the step of performing feature prediction on the masked portion of the masked audio features by the second encoding unit includes:

[0034] Obtain the sequence length of the masked audio features, and generate a relative position index based on the sequence length, wherein the relative position index is the distance index between two sequences in the masked audio features;

[0035] Generate a relative position vector based on the distance index;

[0036] The attention weights are calculated for the relative position vectors, and feature prediction is performed on the masked portion of the masked audio features based on the attention weights to obtain the first training features.

[0037] In some embodiments, the audio encoding module further includes a category mapping unit, and the step of performing feature extraction on the mask training data through the second encoding unit includes:

[0038] The clustering mask features of the mask training data are extracted using the second encoding unit;

[0039] The category mapping unit performs category mapping on the mask features to be clustered to obtain the mask audio features.

[0040] In some embodiments, the audio encoding module further includes a clustering module and a category encoding unit, and the step of performing category mapping on the mask features to be clustered includes:

[0041] The category code is obtained through the category coding unit, and the category code is obtained by the clustering module after performing clustering based on the speech sample data.

[0042] The category mapping unit performs category mapping on the mask features to be clustered according to the category encoding to obtain the mask audio features.

[0043] In some embodiments, the clustering module includes a feature extractor, and before the step of obtaining the category code through the category coding unit, the method further includes:

[0044] Obtain a second training dataset, which includes a preset number of second training data.

[0045] The second training data is input into the clustering module to be trained, so that the third training feature of the second training data can be extracted by the feature extractor;

[0046] Clustering classification is performed on the third training feature to generate the category code.

[0047] Secondly, some embodiments of this application provide a video conversion system based on mask prediction, including an audio extraction module, a text acquisition module, a conversion module, a speech generation module, and an acoustic model;

[0048] The audio extraction module is configured to extract the audio to be converted from the video to be converted, and the audio to be converted includes at least one speaker's audio.

[0049] The text acquisition module is configured to perform speech recognition on the audio to be converted, and obtain audio text and speaker information corresponding to the speaker's audio, wherein the language information of the audio text is a first language;

[0050] The conversion module is configured to convert the audio text to a second language different from the first language to obtain the text to be converted.

[0051] The speech generation module is configured to generate a first audio based on the text to be converted;

[0052] The acoustic model is configured to perform speech feature replacement based on a first audio feature and a second audio feature, and output a target audio; the first audio feature is the audio feature corresponding to a first audio, and the second audio feature is the audio feature corresponding to a second audio; the target audio replaces the audio to be converted to obtain a target video, where the second audio is the audio retrieved based on the speaker information;

[0053] The acoustic model is obtained through training using mask prediction.

[0054] As can be seen from the above technical solutions, this application provides a video conversion method and system based on mask prediction. The method extracts the audio to be converted from the video to be converted; based on the audio to be converted, it obtains the speaker information corresponding to the audio text in a first language and the speaker's audio; and performs language conversion on the audio text according to a second language different from the first language to obtain the text to be converted; it generates a first audio based on the text to be converted; it retrieves a second audio based on the speaker information; it inputs the first and second audio into an acoustic model; and the acoustic model, trained using mask prediction, performs speech feature replacement based on the features of the first and second audio, outputting a target audio; and it replaces the audio to be converted with the target audio to obtain the target video. This application improves the accuracy of speech feature extraction by the acoustic model through mask prediction, thereby improving the accuracy of speech feature extraction from the first and second audio, and enhancing the synthesis effect of the audio after speech conversion. Attached Figure Description

[0055] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 A flowchart illustrating a video conversion method based on mask prediction provided in this application embodiment;

[0057] Figure 2 This is a flowchart illustrating the extraction of target audio features by the audio encoding module in an embodiment of this application.

[0058] Figure 3 This is a flowchart illustrating the audio feature replacement performed by the replacement unit in this embodiment of the application.

[0059] Figure 4 This is a flowchart of the training audio encoding module in an embodiment of this application;

[0060] Figure 5 This is a flowchart illustrating the mask prediction process performed on the first training data in an embodiment of this application.

[0061] Figure 6 This is a flowchart illustrating the mask prediction process performed by the second encoding unit in an embodiment of this application.

[0062] Figure 7 This is a flowchart of the training clustering module in some embodiments of this application. Detailed Implementation

[0063] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0064] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0065] The terms "first," "second," "third," etc., used in the specification and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms can be used interchangeably where appropriate.

[0066] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0067] Users can play videos through smart devices, such as movies, TV series, operas, or variety shows. When watching films from different countries, because the film's language is different from the user's, the user cannot directly understand the meaning of the dialogue based on the characters' words. For example, when a Chinese user watches an English film, if the user cannot understand English, they cannot directly know the meaning of the dialogue based on the audio of the characters' words.

[0068] Therefore, some video resources can include subtitle resources, that is, displaying subtitles in the corresponding language on the screen while the video is playing. For example, when playing a video in English, Chinese subtitles can be added on the screen so that Chinese users can understand the meaning of the dialogue. This requires users to watch the video and subtitles simultaneously to understand the meaning of the characters' dialogue. Although the subtitles are on the screen, repeatedly watching content in different positions affects the user's viewing experience.

[0069] To improve the viewing experience, the dialogue can be translated into other languages ​​before being dubbed in post-production, allowing users to intuitively understand the meaning of the lines based on the dubbing. However, this post-dubbing method is time-consuming and needs to be completed before the video plays, making it impossible to translate and play the dialogue in real time.

[0070] Therefore, speech-to-text conversion technology can be used to convert the language of a film into the user's desired language while preserving the vocal characteristics of the characters, thus achieving real-time translation. To achieve this, the speech-to-text conversion model needs to be trained with a large number of voice samples from specific individuals. However, it is difficult to obtain a large number of voice samples for a particular individual, resulting in low accuracy of speech-to-text conversion in scenarios with few samples.

[0071] To address the issue of low speech conversion accuracy when translating spoken language in scenarios with low sample counts, some embodiments of this application provide a video conversion method based on mask prediction. Figure 1 A flowchart illustrating a video conversion method based on mask prediction provided in an embodiment of this application. See also... Figure 1 The method includes:

[0072] S100: Extract the audio from the video to be converted.

[0073] The video to be converted is the video for which language conversion is to be performed. The audio to be converted includes at least one speaker's audio. For example, if character A in the video says, "The weather is so nice today," then "The weather is so nice today" is the speaker's audio. When there are multiple characters speaking simultaneously in the audio to be converted, character recognition can be performed based on the current video frame. The identified characters are then selected to determine the target character for audio conversion, and their audio is obtained; this is the audio to be converted.

[0074] If it is necessary to perform language conversion on the person audio corresponding to all the people in the current video frame, determine the person audio set corresponding to each person, obtain the person audio set of each person, and perform subsequent language conversion based on the person audio in each person audio set to improve the accuracy when different people speak alternately.

[0075] In some embodiments, the audio to be converted can also be obtained by extracting from the line audio. The line audio can also include the narration audio, that is, the audio that is not directly spoken by the speaker, but is the audio that is dubbed and spliced later. For example, play the audio "Person A is walking in good weather" generated by the film narration in the voice of a preset person, and "Person A is walking in good weather" is the narration audio. Therefore, the audio to be converted can also be extracted from the narration audio.

[0076] S200: Obtain the audio text and the speaker information corresponding to the speaker audio according to the audio to be converted.

[0077] After obtaining the audio to be converted, the audio to be converted can be input into a speech recognition model to perform speech recognition on the audio to be converted through the speech recognition model, so as to obtain the audio text corresponding to the audio to be converted, which is convenient for subsequent language conversion of the audio text.

[0078] In some embodiments, in addition to speech recognition, the audio text can also be obtained by other means. For example, when playing the video to be converted, the subtitle data of the video to be converted can be obtained, and the subtitles can be text-recognized to obtain the audio text corresponding to the audio to be converted. Another example is that the text resource track of the video to be converted can be obtained, and the audio text can be directly obtained through the text resource track. Among them, the video to be converted can include multiple tracks, such as the audio resource track, the video resource track, etc. The text resource track is the text content output by each person or the narration in the video to be converted. Therefore, the video conversion device can obtain the audio text according to the text resource track.

[0079] In some embodiments, it is also necessary to determine the language information of the audio text before performing speech conversion on the corresponding audio text. For this purpose, after the audio text is obtained by speech recognition, the language detection can be performed on the audio text to obtain the language information of the audio text. The language information of the audio text is the first language. For example, when the audio text is "Hello", it can be detected that "Hello" is an English text, so as to determine that the first language is English.

[0080] It should be understood that the Chinese and English languages ​​in the above embodiments are merely illustrative examples. In practical applications, conversion between other languages ​​can also be achieved. Alternatively, conversion between different dialects within the same language can also be implemented, i.e., keeping the audio text unchanged while performing conversion on the audio to be converted using different dialects.

[0081] During the process of speech recognition to convert audio, speaker information corresponding to the speaker's audio can also be obtained, that is, the speaker information of the target person, who is the person who uttered the audio. For example, if a film includes audio of speaker A and speaker B, the speaker information of speaker A and speaker B can be obtained from the audio of speaker A and speaker B, so as to obtain a second audio as a sample based on the speaker information, where both speaker A and speaker B are target persons.

[0082] In a film scene, the speaker's audio is spoken by the actor or crew member playing the character in the film. Therefore, the speaker information is the information of the corresponding actor or crew member. For example, in the film, character A is played by actor C. Therefore, the speaker information can be the information of actor C. In this case, actor C is the target character.

[0083] In some embodiments, the speaker audio for a portion of the film is generated through post-production dubbing; therefore, the speaker audio differs from the actual voices of the actors and crew members portraying the characters in the film. To address this, character recognition can be performed based on the speaker audio to calculate the feature similarity between a preset voice template and the speaker audio, thereby determining the speaker information.

[0084] S300: The audio text is converted to a second language different from the first language to obtain the text to be converted.

[0085] Before performing language conversion on audio text, a second language needs to be determined. The audio text is then converted based on this second language to obtain the text to be converted. This second language information is the language information of the text to be converted. This second language information can be determined by the user. For example, if a user whose native language is Chinese is watching an English movie, and they don't understand English, they cannot understand the meaning of the dialogue based on the characters' spoken English lines. In this case, the first language of the audio text is English, and the user can select Chinese as the second language. The audio text is then converted based on this Chinese language to obtain the text to be converted in Chinese, allowing the user to understand the meaning of the dialogue.

[0086] S400: Generate a first audio based on the text to be converted, and retrieve a second audio based on the speaker information.

[0087] After obtaining the text to be converted, a format conversion can be performed on the text using Text-to-Speech (TTS) technology to generate the first audio. The first audio is the audio of the text to be converted spoken by a preset voice, and the language information of the first audio is the second language.

[0088] However, the first audio at this point is only identical to the text to be converted in terms of text content, but does not match the speaker information in terms of sound features. Therefore, it is necessary to retrieve the second audio based on the speaker information. The second audio is an audio with the same timbre as the speaker information, which will be used as a sample audio for subsequent synthesis of the target audio.

[0089] In some embodiments, the second audio can be obtained from other videos based on speaker information. For example, the audio of the speaker speaking in other videos or in public can be obtained based on speaker information. To improve the extraction accuracy of the second audio, after the second audio is extracted, background noise removal and noise reduction processing can be performed on the second audio to reduce the interference of noise and background sounds on the human voice in the second audio.

[0090] S500: Input the first audio and the second audio into the acoustic model, and perform speech feature replacement based on the first audio features and the second audio features through the acoustic model to output the target audio;

[0091] The acoustic model is trained using a mask prediction method. Mask prediction represents the process of masking at least a portion of the training data and then predicting the results from the masked training samples during the acoustic model training process. This mask prediction training method improves the acoustic model's ability to understand speech features within the initial audio features, thereby increasing the efficiency and accuracy of subsequent speech feature replacement and enhancing the precision of the target audio synthesis.

[0092] The first audio feature is the audio feature corresponding to the first audio file. Since the first audio file is generated based on the text to be converted, the first audio feature is used to represent the speaker's voice features in the first audio file. The second audio feature is the audio feature corresponding to the second audio file, used to represent the voice features of the target person in the second audio file.

[0093] After obtaining the first and second audio files, they can be input into the acoustic model, which will extract the first audio features from the first audio file. Following the extraction of these features, the acoustic model will perform speech feature replacement on them, replacing the speaker's voice features in the first audio file with the target person's voice features, thereby generating the target audio file containing the target person's voice features speaking the text to be converted.

[0094] S600: Replace the audio to be converted with the target audio to obtain the target video.

[0095] After obtaining the target audio, the audio to be converted is replaced with the target audio, so that in the target video, the language information of the audio to be converted is switched from the first language to the second language, thus realizing the conversion of the audio of the dialogue in the video.

[0096] In some embodiments, the acoustic model includes an audio encoding module and an audio decoding module. For example... Figure 2 As shown, the audio encoding module includes a feature encoding unit and a permutation unit. The feature encoding unit is configured to extract first audio features from a first audio source and second audio features from a second audio source. The permutation unit is configured to perform speech feature replacement on the first and second audio features according to a permutation network to obtain target audio features. The audio decoding module is configured to decode the target audio features to obtain the target audio.

[0097] During the process of speech feature replacement performed by the permutation unit, such as Figure 3 As shown, the acoustic model can calculate feature similarity values ​​through a permutation unit. These similarity values ​​are the similarity between the second audio feature and the first audio feature. Furthermore, the permutation unit, based on the KNN algorithm, queries a third audio feature within the second audio feature based on the feature similarity value. This third audio feature is then used to replace the first audio feature according to the permutation network, resulting in the target audio feature. The third audio feature is the second audio feature whose feature similarity value is greater than or equal to a similarity threshold. By replacing the original first audio feature with a third audio feature similar to it, the acoustic features of the obtained target audio feature better match the acoustic features of the target person, thereby improving the acoustic similarity of the target audio.

[0098] In some embodiments, the feature encoding unit extracts audio features from the audio through feature encoding. For example, the feature encoding unit performs feature encoding on the first audio to obtain the first audio features as follows: Then, feature encoding is performed on the second audio to obtain the second audio features. Where, x∈R d For the feature representation of a frame, N s N is the length of the first audio signal. t The length of the second audio signal.

[0099] During the calculation of feature similarity values, the permutation unit can generate a fourth audio feature based on the first audio feature using a non-nearest neighbor substitution method. The fourth audio feature is... After generating the fourth audio feature, the permutation unit can use a proximity algorithm to calculate the proximity values ​​between the second and fourth audio features, and generate feature similarity values ​​based on these proximity values. Then, based on these feature similarity values, a third audio feature is determined from the second audio features. The third audio feature is X. t,e =X t ∪X e During the speech feature replacement process, the replacement unit can perform feature sequence X of the third audio feature. t,e Averaging is performed to obtain the speech feature sequence to be replaced, and the feature sequence X in the first audio feature is replaced by the speech sequence to be replaced. s This leads to the final target audio encoding, i.e.

[0100] In some embodiments, for the audio encoding module to perform the aforementioned audio synthesis effect of speech feature replacement, a specific training process needs to be performed on the acoustic model. To this end, before inputting the first and second audio files into the acoustic model, a first training dataset needs to be obtained. The first training dataset includes a predetermined amount of first training data, which is audio data used to train the audio encoding module. The first training data includes at least one speaker's audio, in order to train the audio encoding module's ability to extract audio features.

[0101] Before training the audio coding module, its parameters need to be initialized, i.e., some parameters of the feature coding unit and the permutation unit are randomly initialized. After initialization, the first training data can be input into the audio coding module to be trained, so that the first training features of the first training data can be extracted through the feature coding unit. Since the audio coding module needs to perform speech feature replacement between different audio features, during the training process, reference training data different from the first training data can be input into the audio coding module as sample audio data for speech feature replacement. Thus, speech feature replacement is performed through the first training features and the reference training features extracted from the reference training data to complete the training process of the permutation unit.

[0102] In some embodiments, such as Figure 4As shown, two first training features from different first training data can also be input into the permutation unit to perform speech feature replacement on the two first training features, resulting in a second training feature. After obtaining the second training feature, the acoustic model calculates the feature loss between the second training feature and the audio feature label using a loss function. The audio feature label represents the standard audio features after speech feature replacement and is used to determine the convergence degree of the audio coding module. If the feature loss is less than or equal to the first loss threshold, it indicates that the feature accuracy of the second training feature output by the audio coding module after speech feature replacement reaches the accuracy of the audio feature label. The acoustic model can then output the audio coding module based on the current parameters of the feature coding unit and the permutation unit. If the feature loss is greater than the first loss threshold, it indicates that the feature accuracy of the second training feature output by the audio coding module after speech feature replacement has not reached the accuracy of the audio feature label. In this case, it is necessary to iteratively train the coding unit and the permutation unit using the first training data, continuously updating the parameters of the training coding unit and the permutation unit during the training process until the calculated feature loss is less than or equal to the first loss threshold, thus completing the training process of the audio coding module.

[0103] In some embodiments, to improve the accuracy of audio feature extraction by the feature coding unit, the feature coding unit may also employ mask prediction to extract audio features. Therefore, as... Figure 5 As shown, the feature encoding unit includes a masking subunit, a first encoding unit, and a second encoding unit. The masking subunit is configured to mask a portion of the audio features of the first training data to facilitate subsequent feature prediction based on the masked audio features. To this end, after the first training data is input into the feature encoding unit, the masking subunit can perform masking processing on the first training data to obtain masked training data. The masking subunit can perform masking on the first training data according to a preset masking ratio, which represents the proportion of data to be masked. The masking ratio can be between 30% and 50%. For example, when the masking ratio is 50%, half of the first training data is masked. The masked first training data can be a single data portion; for example, the first 50% is masked, and the last 50% is not, so that prediction can be performed on the first 50% of the audio features based on the last 50% of the first training data. The first training data that is masked can also be divided into multiple data portions. For example, the first 25% of the first training data is masked, the second 25% is not masked, the third 25% is masked, and the fourth 25% is not masked. This is to facilitate performing predictions on the masked data portions based on the first training data that is not masked.

[0104] After completing the masking process, the masking sub-unit can output the masking training data and input it into the first encoding unit. The first encoding unit is a CNN encoding unit, which can perform feature extraction on the masking training data using a convolutional neural network with a pre-defined kernel size, and output the masked audio features.

[0105] Since some training data in the masked training data is masked, the first encoding unit cannot extract features from the masked data. Instead, the first encoding unit can extract audio features from the unmasked portion of the masked training data to obtain the masked audio features.

[0106] The first masking unit can input the masked audio features into the second coding unit, which can be a Transformer coding unit. The second coding unit can further extract deeper features from the masked audio features and perform feature prediction on the masked part of the masked audio features based on the extracted deeper features to obtain the first training features. Thus, by training with the mask, the accuracy of the feature coding unit in extracting audio features is improved.

[0107] During feature prediction by the second coding unit, prediction needs to be performed based on the attention weights of different feature sequences to improve the accuracy of feature prediction. Therefore, the second coding unit also needs to calculate the attention weights of the feature sequences of the masked audio features before performing feature prediction.

[0108] Figure 6 This is a flowchart illustrating the calculation of attention weights based on mask training features in an embodiment of this application. See also... Figure 6 The mask training features include multiple mask feature sequences. The second encoding unit can obtain the sequence length of the mask audio features and generate a relative position index based on the sequence length. The relative position index is the distance index between any two sequences in the mask audio features. For example, for mask feature sequence i, the second encoding unit can use the relative distance index r(i, j) between mask feature sequence i and mask feature sequence j. After generating the relative distance index, the second encoding unit can generate a relative position vector between mask feature sequence i and mask feature sequence j based on the distance index, and calculate the attention weight of mask feature sequence i relative to mask feature sequence j based on the relative position vector. It should be understood that the above embodiment is the process of calculating the attention weight of mask feature sequence i relative to mask feature sequence j. In the actual training process, attention weights can also be calculated based on any different mask feature sequences. The second encoding unit can perform feature prediction on the masked part of the mask audio features based on the attention weights of multiple sets of mask feature sequences to obtain the first training features.

[0109] This application combines the relative position information of the mask feature sequences with the calculation of attention weights during the attention weight calculation process to achieve flexible control over the information interaction between different mask feature sequences, improve the feature correlation between different mask feature sequences, and thus enhance the accuracy of prediction.

[0110] To improve the accuracy of audio extraction, clustering extraction can be used during the extraction of training data. Furthermore, clustering extraction can be combined with mask training.

[0111] Therefore, the second masking unit can also extract mask audio features using clustering. To this end, the audio encoding module also includes a category mapping unit, which is configured to perform category mapping on the audio features. During feature extraction from the mask training data by the second encoding unit, the second encoding unit can extract the mask features to be clustered from the mask training data. The mask features to be clustered are the mask audio features that have not undergone clustering. When the mask training data includes multiple speakers, the mask features to be clustered can include the audio features of multiple speakers; however, the audio features of other speakers will affect the extraction accuracy of the mask audio features.

[0112] To address this, the category mapping unit can perform category mapping on the mask features to be clustered based on a preset category code. This allows for classification of the mask features according to different speakers, reducing the impact of other speakers' audio on the extraction of mask audio features. The category mapping unit consists of a mapping layer used to perform category mapping on the mask features to be clustered based on the category code, thereby extracting mask audio features from the classified mask features and improving the extraction accuracy of mask audio features.

[0113] The audio encoding module also includes a clustering module and a category encoding unit. The clustering module provides category codes to the audio encoding module during training. During training, the category encoding unit obtains the category codes obtained by the clustering module after performing clustering on the speech sample data, and then provides these category codes to the category mapping unit. After obtaining the category codes, the category mapping unit performs category mapping on the mask features to be clustered according to the category codes, thus clustering the mask features to obtain the mask audio features.

[0114] It should be noted that the category coding unit does not participate in the feature extraction process of the audio coding module during application. It only introduces category coding into the audio coding module during the training process to train the clustering extraction function of the audio coding module.

[0115] In the actual training process, in addition to training and updating the parameters of the feature encoding unit, this embodiment also trains the audio encoding module based on the category encoding corresponding to the training data and the real category encoding ID obtained by the clustering module to improve the accuracy of audio feature extraction. Therefore, the acoustic model also needs to train the clustering module. The clustering module may include a feature extractor configured to extract audio features from the training data. During the training of the clustering module, a second training dataset needs to be obtained. The second training dataset includes a preset number of second training data points, which may be the same as or different from the first training data. The second training data may be based on LibriSpeech-960 and AISHELL-3 data; specifically, it obtains speech sample data from 200 speakers, with 200 clusters.

[0116] The second training data is input into the clustering module to be trained. A feature extractor extracts the third training features from the second training data and performs clustering classification on these third training features to generate category codes. During the clustering classification process, the feature extractor can calculate feature similarity on the third training features and cluster the third training data whose feature similarity is less than or equal to a similarity threshold, i.e., clustering similar audio features. After clustering, audio features of the same category can represent the audio features of an independent person, while audio features of different categories differ significantly. This facilitates the extraction of audio features for each person based on category features, thereby improving the feature extraction effect.

[0117] To improve the clustering training effect, the clustering module may further include a first clustering module and a second clustering module. The first and second clustering modules can respectively perform clustering training on the audio data of different individuals in the second training data. The feature extractors of the first and second clustering modules can be any two of three types of feature extractors: SoftHubert, Hubert, and WAV2VEC2.0. For example, the first clustering model can use Hubert as its feature extractor, and the second clustering model can use WAV2VEC2.0 as its feature extractor.

[0118] Therefore, the first and second clustering modules can be trained simultaneously using the second training data, enabling them to classify audio data from different speakers within the second training data. Since the first and second clustering modules employ different feature extractors, they can cluster from different dimensions, improving the extraction of various audio features, such as timbre and prosody. Thus, the combination of different feature extraction methods from the first and second clustering modules can enrich the training results when performing language transformations subsequently.

[0119] In some embodiments, such as Figure 7 As shown, during the training process of the first and second clustering modules, it is necessary to encode the categories output by the first and second clustering modules. For example, after the first clustering module clusters the second training data, it obtains different feature categories, which can be assigned ID1.1, ID1.2, ..., ID1.9, etc. Similarly, after the second clustering model clusters the second training data, it obtains different feature categories, which can be assigned ID2.1, ID2.2, ..., ID2.9, etc. The purpose of category encoding is to ensure that each category after clustering by the clustering module has a unique identifier for differentiation, so as to facilitate category mapping and encoding during subsequent language transformation training.

[0120] In some embodiments, the acoustic model can also update the loss function of the audio coding module based on the first clustering module and the second clustering module. For example, the average cross-entropy between the true class code of the first clustering model and the predicted class code of the audio coding module is minimized, while the average cross-entropy between the true class code of the second clustering model and the predicted class code of the audio coding module is also minimized. Based on this, the loss function is updated, and the parameters of the audio coding module are updated. Through the above training method, the ability of the audio coding module to classify timbre categories is further enhanced.

[0121] To facilitate the execution of the video conversion method based on mask prediction described in the above specification, some embodiments of this application also provide a video conversion system based on mask prediction, including an audio extraction module, a text acquisition module, a conversion module, a speech generation module, and an acoustic model;

[0122] The audio extraction module is configured to extract the audio to be converted from the video to be converted, and the audio to be converted includes at least one speaker's audio.

[0123] The text acquisition module is configured to perform speech recognition on the audio to be converted, and obtain audio text and speaker information corresponding to the speaker's audio, wherein the language information of the audio text is a first language;

[0124] The conversion module is configured to convert the audio text to a second language different from the first language to obtain the text to be converted.

[0125] The speech generation module is configured to generate a first audio based on the text to be converted;

[0126] The acoustic model is configured to perform speech feature replacement based on a first audio feature and a second audio feature, and output a target audio; the first audio feature is the audio feature corresponding to a first audio, and the second audio feature is the audio feature corresponding to a second audio; the target audio replaces the audio to be converted to obtain a target video, where the second audio is the audio retrieved based on the speaker information; the acoustic model is obtained through training by mask prediction.

[0127] As can be seen from the above technical solution, this application extracts the audio to be converted from the video to be converted, obtains the speaker information corresponding to the audio text in the first language and the speaker's audio based on the audio to be converted, and performs language conversion on the audio text according to a second language different from the first language to obtain the text to be converted; generates the first audio based on the text to be converted, retrieves the second audio based on the speaker information, inputs the first and second audio into an acoustic model, and performs speech feature replacement based on the features of the first and second audios using the acoustic model trained by mask prediction, outputting the target audio, and replacing the audio to be converted with the target audio to obtain the target video. This application improves the accuracy of the acoustic model in extracting speech features by using mask prediction, thereby improving the accuracy of extracting speech features from the first and second audios and enhancing the synthesis effect of the audio after speech conversion.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0129] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the foregoing exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be made based on the foregoing teachings. The selection and description of the above embodiments are for the purpose of better explaining the contents of this disclosure, thereby enabling those skilled in the art to better utilize the described embodiments.

Claims

1. A video conversion method based on mask prediction, characterized in that, include: Extract the audio to be converted from the video to be converted, wherein the audio to be converted includes at least one speaker's audio; Based on the audio to be converted, audio text and speaker information corresponding to the speaker's audio are obtained, wherein the language information of the audio text is the first language; The audio text is converted to a second language, which is different from the first language, to obtain the text to be converted. A first audio is generated based on the text to be converted, and a second audio is retrieved based on the speaker information; The first audio and the second audio are input into an acoustic model, which includes an audio encoding module and an audio decoding module. The audio encoding module includes a feature encoding unit and a permutation unit. The feature encoding unit includes a mask subunit, a first encoding unit, and a second encoding unit. The first audio feature of the first audio is extracted by the feature encoding unit, and the second audio feature of the second audio is extracted by the feature encoding unit; the first audio feature is the voice feature of the speaker in the first audio. The permutation unit calculates a feature similarity value, which is the similarity value between the second audio feature and the first audio feature. Based on the feature similarity value, a third audio feature is extracted from the second audio feature, and the first audio feature is replaced by the third audio feature to obtain the target audio feature. The third audio feature is the voice feature of the target person. The third audio feature is a second audio feature with a feature similarity value greater than or equal to a similarity threshold. The target audio feature is decoded by the audio decoding module to output the target audio. The acoustic model is obtained through training using mask prediction. The mask prediction is used to represent masking at least a portion of the training data and to predict the masked training samples during training. Replace the audio to be converted with the target audio to obtain the target video; Before the step of inputting the first audio and the second audio into the acoustic model, the method further includes: masking a portion of the audio features of the first training data by a masking subunit, and performing masking processing on the first training data through the masking subunit to obtain masked training data; The first encoding unit performs feature extraction on the mask training data to obtain the mask audio features; The second encoding unit performs feature prediction on the masked portion of the masked audio features to obtain a first training feature; wherein, the first training feature is used to train the permutation unit, and speech feature replacement is performed using the first training feature and the reference training feature extracted from the reference training data.

2. The video conversion method based on mask prediction according to claim 1, characterized in that, The step of calculating feature similarity values ​​through the permutation unit includes: The permutation unit generates a fourth audio feature based on the first audio feature, and the fourth audio feature is generated according to the non-nearest neighbor replacement algorithm. According to the proximity algorithm, the proximity value between the second audio feature and the fourth audio feature is calculated; Feature similarity values ​​are generated based on the neighbor values.

3. The video conversion method based on mask prediction according to claim 1, characterized in that, Before the step of inputting the first audio and the second audio into the acoustic model, the method further includes: Obtain a first training dataset, which includes a preset number of first training data. The first training data is input into the audio encoding module to be trained, so as to extract the first training feature of the first training data through the feature encoding unit. At least two first training features are input into the permutation unit to perform speech feature replacement on the first training features to obtain second training features; Calculate the feature loss between the second training feature and the audio feature label, where the audio feature label is used to characterize standard audio features; If the feature loss is less than or equal to the first loss threshold, the audio encoding module is output according to the current parameters of the feature encoding unit and the current parameters of the permutation unit; If the feature loss is greater than the first loss threshold, then the feature encoding unit and the permutation unit are iteratively trained based on the first training data.

4. The video conversion method based on mask prediction according to claim 1, characterized in that, The step of performing feature prediction on the masked portion of the masked audio features through the second encoding unit includes: Obtain the sequence length of the masked audio features, and generate a relative position index based on the sequence length, wherein the relative position index is the distance index between two sequences in the masked audio features; Generate a relative position vector based on the distance index; The attention weights are calculated for the relative position vectors, and feature prediction is performed on the masked portion of the masked audio features based on the attention weights to obtain the first training features.

5. The video conversion method based on mask prediction according to claim 1, characterized in that, The audio encoding module further includes a category mapping unit, and the step of performing feature extraction on the mask training data through the second encoding unit includes: The clustering mask features of the mask training data are extracted using the second encoding unit; The category mapping unit performs category mapping on the mask features to be clustered to obtain the mask audio features.

6. The video conversion method based on mask prediction according to claim 5, characterized in that, The audio encoding module further includes a clustering module and a category encoding unit. The step of performing category mapping on the mask features to be clustered includes: The category code is obtained through the category coding unit, and the category code is obtained by the clustering module after performing clustering based on the speech sample data. The category mapping unit performs category mapping on the mask features to be clustered according to the category encoding to obtain the mask audio features.

7. The video conversion method based on mask prediction according to claim 6, characterized in that, The clustering module includes a feature extractor, and before the step of obtaining the category code through the category encoding unit, the method further includes: Obtain a second training dataset, which includes a preset number of second training data. The second training data is input into the clustering module to be trained, so that the third training feature of the second training data can be extracted by the feature extractor; Clustering classification is performed on the third training feature to generate the category code.

8. A video conversion system based on mask prediction, characterized in that, It includes an audio extraction module, a text acquisition module, a conversion module, a speech generation module, and an acoustic model; The audio extraction module is configured to extract the audio to be converted from the video to be converted, and the audio to be converted includes at least one speaker's audio. The text acquisition module is configured to obtain audio text and speaker information corresponding to the speaker's audio based on the audio to be converted, wherein the language information of the audio text is a first language; The conversion module is configured to convert the audio text to a second language different from the first language to obtain the text to be converted. The speech generation module is configured to generate a first audio based on the text to be converted, and to retrieve a second audio based on the speaker information; The acoustic model is configured to input the first audio and the second audio into the acoustic model. The acoustic model includes an audio encoding module and an audio decoding module. The audio encoding module includes a feature encoding unit and a permutation unit. The feature encoding unit includes a mask subunit, a first encoding unit, and a second encoding unit. The first audio feature of the first audio is extracted by the feature encoding unit, and the second audio feature of the second audio is extracted by the feature encoding unit; the first audio feature is the voice feature of the speaker in the first audio. The permutation unit calculates a feature similarity value, which is the similarity value between the second audio feature and the first audio feature. Based on the feature similarity value, a third audio feature is extracted from the second audio feature, and the first audio feature is replaced by the third audio feature to obtain a target audio feature. The third audio feature is the voice feature of the target person; the third audio feature is a second audio feature with a feature similarity value greater than or equal to a similarity threshold. The target audio feature is decoded by an audio decoding module to output the target audio. The target audio is then used to replace the audio to be converted to obtain the target video. The mask prediction is used to characterize the masking of at least a portion of the training data and to predict the masked training samples during training; the acoustic model is obtained through training by mask prediction. Specifically, before inputting the first audio and the second audio into the acoustic model, the masking subunit masks some audio features of the first training data, and the first training data is masked by the masking subunit to obtain masked training data. The first encoding unit performs feature extraction on the mask training data to obtain the mask audio features; The second encoding unit performs feature prediction on the masked portion of the masked audio features to obtain a first training feature; wherein, the first training feature is used to train the permutation unit, and speech feature replacement is performed using the first training feature and the reference training feature extracted from the reference training data.

Images