Video conversion method and system based on mask prediction

Abstract

The invention provides a video conversion method and system based on mask prediction, and the method comprises the steps: extracting a to-be-converted audio of a to-be-converted video, obtaining an audio text of a first language and speaker information corresponding to the speaker audio according to the to-be-converted audio, and carrying out the language conversion of the audio text according to a second language different from the first language. Obtaining a to-be-converted text; the method comprises the following steps: generating a first audio according to a to-be-converted text, retrieving a second audio according to speaker information, inputting the first audio and the second audio into an acoustic model, executing voice feature replacement according to a first audio feature and a second audio feature through the acoustic model obtained through a mask prediction training mode, and outputting a target audio, and replacing the to-be-converted audio with the target audio to obtain a target video. Through the mask prediction mode, the accuracy of voice feature extraction of the acoustic model is improved, so that the accuracy of voice feature extraction of the first audio and the second audio is improved, and the synthesis effect of the audio after voice translation is improved.

Description

[0001] This application claims priority to a Chinese patent application filed with the State Intellectual Property Office with application number 202311479579.X on November 8, 2023. The entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the technical field of speech conversion, and in particular to a video conversion method and system based on mask prediction. Background Art

[0003] When users watch movies from different countries, they cannot directly understand the meaning of the lines from the audio of the lines spoken by the characters in the movie because the language of the movie is different from the user's language. Therefore, users need to watch subtitles continuously while watching the movie to understand the meaning of the lines of the characters in the movie.

[0004] In order to improve the viewing experience of the film, the lines in the film can be translated into different languages ​​and then dubbed, so that users can intuitively understand the semantics of the lines according to the dubbing of the lines. However, this post-dubbing method consumes a lot of time and needs to be completed before playing the film, and it is impossible to translate and play the lines in real time.

[0005] To this end, voice conversion technology can be used to convert the language of the movie into the language desired by the user while retaining the voice characteristics of the characters in the movie, thereby achieving the effect of real-time translation. In order to achieve the above effect, a large number of voice samples of the characters need to be used to train the voice conversion model. However, it is difficult to obtain a large number of voice samples of specific characters, resulting in low accuracy of voice conversion in scenarios with low samples. Summary of the invention

[0006] In order to solve the problem of low voice conversion accuracy when translating human language in a low sample scenario, some embodiments of the present application provide a video conversion method and system based on mask prediction.

[0007] In a first aspect, some embodiments of the present application provide a video conversion method based on mask prediction, comprising:

[0008] Extracting audio to be converted from the video to be converted, wherein the audio to be converted includes at least one speaker audio;

[0009] According to the audio to be converted, an audio text and speaker information corresponding to the speaker audio are obtained, wherein the language information of the audio text is a first language;

[0010] Performing language conversion on the audio text according to a second language different from the first language to obtain a text to be converted;

[0011] generating a first audio according to the text to be converted, and retrieving a second audio according to the speaker information;

[0012] The first audio and the second audio are input into an acoustic model, so as to perform speech feature replacement according to the first audio feature and the second audio feature through the acoustic model, and output a target audio; the first audio feature is an audio feature corresponding to the first audio, and the second audio feature is an audio feature corresponding to the second audio; the acoustic model is obtained by a mask prediction training method; wherein the mask prediction is used to characterize mask processing of at least part of the training data, and to predict the training samples after mask processing during the training process;

[0013] The target audio replaces the audio to be converted to obtain the target video.

[0014] In some embodiments, the acoustic model includes an audio encoding module and an audio decoding module, the audio encoding module includes a feature encoding unit and a replacement unit, and after the step of inputting the first audio and the second audio into the acoustic model, the method further includes:

[0015] extracting a first audio feature of the first audio by the feature encoding unit, and extracting a second audio feature of the second audio by the feature encoding unit;

[0016] The replacement unit calculates a feature similarity value, which is a similarity value between the second audio feature and the first audio feature, and extracts a third audio feature from the second audio feature according to the feature similarity value, and replaces the first audio feature according to the third audio feature to obtain a target audio feature; the third audio feature is a second audio feature whose feature similarity value is greater than or equal to a similarity threshold;

[0017] The target audio feature is decoded by the audio decoding module to obtain the target audio.

[0018] In some embodiments, the step of calculating the feature similarity value by the replacement unit includes:

[0019] Generate a fourth audio feature according to the first audio feature by the replacement unit, wherein the fourth audio feature is generated according to a non-contiguous replacement algorithm;

[0020] Calculating a proximity value between the second audio feature and the fourth audio feature according to a proximity algorithm;

[0021] A feature similarity value is generated based on the proximity value.

[0022] In some embodiments, before the step of inputting the first audio and the second audio into the acoustic model, the method further includes:

[0023] Acquire a first training data set, where the first training data set includes a preset amount of first training data;

[0024] Inputting the first training data into an audio encoding module to be trained, so as to extract a first training feature of the first training data through a feature encoding unit;

[0025] Inputting at least two first training features into the replacement unit to perform speech feature replacement on the first training features to obtain second training features;

[0026] Calculating a feature loss between the second training feature and an audio feature label, wherein the audio feature label is used to characterize a standard audio feature;

[0027] If the feature loss is less than or equal to a first loss threshold, outputting the audio encoding module according to current parameters of the feature encoding unit and current parameters of the replacement unit;

[0028] If the feature loss is greater than a first loss threshold, the feature encoding unit and the replacement unit are iteratively trained according to first training data.

[0029] In some embodiments, the feature encoding unit includes a mask subunit, a first encoding unit, and a second encoding unit, and the step of extracting the first training feature of the first training data through the feature encoding unit includes:

[0030] Inputting the first training data into the feature encoding unit so as to perform mask processing on the first training data through the mask subunit to obtain masked training data;

[0031] Performing feature extraction on the mask training data by the first encoding unit to obtain mask audio features;

[0032] The second encoding unit performs feature prediction on the masked part of the masked audio feature to obtain a first training feature.

[0033] In some embodiments, the step of performing feature prediction on the masked portion of the masked audio feature by the second encoding unit includes:

[0034] Obtaining a sequence length of the masked audio feature, and generating a relative position index according to the sequence length, wherein the relative position index is a distance index between two sequences in the masked audio feature;

[0035] Generate a relative position vector according to the distance index;

[0036] An attention weight is calculated for the relative position vector, and a feature prediction is performed on the masked portion of the masked audio feature according to the attention weight to obtain a first training feature.

[0037] In some embodiments, the audio encoding module further includes a category mapping unit, and the step of performing feature extraction on the mask training data by the second encoding unit includes:

[0038] Extracting, by the second encoding unit, mask features to be clustered of the mask training data;

[0039] The category mapping unit performs category mapping on the mask features to be clustered to obtain the mask audio features.

[0040] In some embodiments, the audio encoding module further includes a clustering module and a category encoding unit, and the step of performing category mapping on the mask features to be clustered includes:

[0041] Acquiring a category code through the category coding unit, wherein the category code is obtained by clustering the speech sample data by the clustering module;

[0042] The category mapping unit performs category mapping on the mask features to be clustered according to the category codes to obtain the mask audio features.

[0043] In some embodiments, the clustering module includes a feature extractor, and before the step of obtaining the category code by the category coding unit, the method further includes:

[0044] Acquire a second training data set, where the second training data set includes a preset amount of second training data;

[0045] Inputting the second training data into a clustering module to be trained, so as to extract a third training feature of the second training data through the feature extractor;

[0046] Cluster classification is performed on the third training feature to generate the category code.

[0047] In a second aspect, some embodiments of the present application provide a video conversion system based on mask prediction, including an audio extraction module, a text acquisition module, a conversion module, a speech generation module and an acoustic model;

[0048] The audio extraction module is configured to extract the audio to be converted of the video to be converted, wherein the audio to be converted includes at least one speaker audio;

[0049] The text acquisition module is configured to speech recognize the audio to be converted, obtain an audio text and speaker information corresponding to the speaker audio, and the language information of the audio text is the first language;

[0050] The conversion module is configured to perform language conversion on the audio text according to a second language different from the first language to obtain a text to be converted;

[0051] The speech generation module is configured to generate a first audio according to the text to be converted;

[0052] The acoustic model is configured to perform speech feature replacement according to a first audio feature and a second audio feature, and output a target audio; the first audio feature is an audio feature corresponding to the first audio, and the second audio feature is an audio feature corresponding to the second audio; the target audio is replaced with the audio to be converted to obtain a target video, and the second audio is the audio retrieved according to the speaker information;

[0053] The acoustic model is obtained through a mask prediction training method.

[0054] It can be seen from the above technical scheme that the present application provides a video conversion method and system based on mask prediction, wherein the method extracts the audio to be converted of the video to be converted, obtains the speaker information corresponding to the audio text of the first language and the speaker audio according to the audio to be converted, and performs language conversion on the audio text according to a second language different from the first language to obtain the text to be converted; generates a first audio according to the text to be converted, retrieves a second audio according to the speaker information, inputs the first audio and the second audio into an acoustic model, performs speech feature replacement according to the first audio feature and the second audio feature using the acoustic model obtained by training through mask prediction, outputs a target audio, replaces the audio to be converted with the target audio, and obtains a target video. The present application improves the accuracy of extracting speech features of the acoustic model by means of mask prediction, thereby improving the accuracy of extracting speech features of the first audio and the second audio, and improving the synthesis effect of the audio after speech conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0056] Figure 1 A flowchart of a video conversion method based on mask prediction provided in an embodiment of the present application;

[0057] Figure 2 This is a flowchart of the audio encoding module extracting target audio features in an embodiment of the present application;

[0058] Figure 3 This is a flowchart of the replacement unit performing audio feature replacement in an embodiment of the present application;

[0059] Figure 4 This is a flowchart of training an audio encoding module in an embodiment of the present application;

[0060] Figure 5 This is a flowchart of performing mask prediction on first training data in an embodiment of the present application;

[0061] Figure 6 This is a flowchart of the second encoding unit performing mask prediction in an embodiment of the present application;

[0062] Figure 7 This is a flowchart of training a clustering module in some embodiments of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0064] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0065] The terms "first", "second", "third", etc. in the specification and the above drawings of this application are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise specified. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0066] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0067] Users can play videos on smart devices, such as movies, TV dramas, operas or variety shows. When users watch movies from different countries, they cannot directly understand the semantics of the lines spoken by the characters in the movie because the language of the movie is different from the user's language. For example, when a Chinese user watches an English movie, if the user cannot understand English, he cannot directly know the semantics of the lines based on the audio of the English lines spoken by the characters in the movie.

[0068] To this end, some movie resources may include subtitle resources, that is, when playing the movie screen, subtitles in the corresponding language are displayed in the movie screen. For example, when playing a movie in English, Chinese subtitles are added to the movie screen so that Chinese users can understand the meaning of the lines in the movie through Chinese subtitles. This requires users to watch the screen and the subtitles while watching the movie so that they can know the semantics of the characters' lines in the movie. Although the subtitles are located in the movie screen, repeatedly watching content located in different positions affects the user's viewing experience.

[0069] In order to improve the viewing experience of the film, the lines in the film can be translated into different languages ​​and then dubbed, so that users can intuitively understand the semantics of the lines according to the dubbing of the lines. However, this post-dubbing method consumes a lot of time and needs to be completed before playing the film, and it is impossible to translate and play the lines in real time.

[0070] To this end, voice conversion technology can be used to convert the language of the movie into the language desired by the user while retaining the voice characteristics of the characters in the movie, thereby achieving the effect of real-time translation. In order to achieve the above effect, a large number of voice samples of the characters need to be used to train the voice conversion model. However, it is difficult to obtain a large number of voice samples of specific characters, resulting in low accuracy of voice conversion in scenarios with low samples.

[0071] In order to solve the problem of low voice conversion accuracy when translating human language in a low sample scenario, some embodiments of the present application provide a video conversion method based on mask prediction. Figure 1 A flowchart of a video conversion method based on mask prediction provided by an embodiment of the present application. Figure 1 , the method comprising:

[0072] S100: extracting the audio to be converted from the video to be converted.

[0073] The video to be converted is a video to be converted into a language, and the audio to be converted includes at least one speaker audio. For example, character A in the film says "It's a nice day today", where "It's a nice day today" is the speaker audio. When there are multiple characters speaking at the same time in the audio to be converted, character recognition can be performed based on the current video screen, and selection can be made based on the recognized characters to determine the target character for the audio conversion, and the character audio of the target character, i.e., the audio to be converted, can be obtained.

[0074] If it is necessary to convert the language of the character audio corresponding to all the characters in the current video screen, determine the character audio set corresponding to each character, obtain the character audio set of each character, and perform subsequent language conversion based on the character audio in each character audio set to improve the accuracy when different characters speak in an alternating manner.

[0075] In some embodiments, the audio to be converted can also be extracted from the line audio. The line audio can also include narration audio, that is, audio that is not spoken directly by the speaker, but is dubbed and spliced ​​in the later stage. For example, the audio "Character A is walking in good weather" generated by the narration of the film with the timbre of the preset character is played, and "Character A is walking in good weather" is the narration audio. Therefore, the audio to be converted can also be extracted from the narration audio.

[0076] S200: Obtaining an audio text and speaker information corresponding to the speaker audio according to the audio to be converted.

[0077] After obtaining the audio to be converted, the audio to be converted can be input into a speech recognition model to perform speech recognition on the audio to be converted through the speech recognition model to obtain an audio text corresponding to the audio to be converted, so as to facilitate subsequent language conversion of the audio text.

[0078] In some embodiments, in addition to speech recognition, audio text can also be obtained by other means. For example, when playing the video to be converted, the subtitle data of the video to be converted can be obtained, and the subtitles can be recognized to obtain the audio text corresponding to the audio to be converted. For another example, the text resource track of the video to be converted can also be obtained, and the audio text can be directly obtained through the text resource track. Among them, the video to be converted can include multiple tracks, such as audio resource tracks, video resource tracks, etc. The text resource track is the text content output by each character or narration in the video to be converted. Therefore, the video conversion device can obtain the audio text according to the text resource track.

[0079] In some embodiments, it is also necessary to determine the language information of the audio text before performing speech conversion on the corresponding audio text. To this end, after the audio text is obtained by speech recognition, language detection can be performed on the audio text to obtain the language information of the audio text. Among them, the language information of the audio text is the first language. For example, when the audio text is "Hello", "Hello" can be detected as Chinese text, thereby determining that the first language is Chinese. For another example, when the audio text is "Hello", "Hello" can be detected as English text, thereby determining that the first language is English.

[0080] It should be understood that the Chinese and English languages ​​in the above embodiments are only exemplary, and in actual application, conversion between other different languages ​​can also be achieved. Alternatively, conversion between different dialects in the same language can also be achieved, that is, the audio text remains unchanged, and the conversion audio is performed through different dialects.

[0081] In the process of speech recognition of the audio to be converted, speaker information corresponding to the speaker audio, that is, speaker information of the target person, can also be obtained. The target person is the person who speaks the speaker audio. For example, when a movie includes speaker audio of person A and speaker audio of person B, speaker information of person A and speaker information of person B can be obtained based on speaker audio of person A and speaker audio of person B, so as to obtain a second audio as a sample based on the speaker information, wherein both person A and person B are target persons.

[0082] In a movie scene, the speaker audio is spoken by the cast member who plays the character in the movie, so the speaker information is the information of the corresponding cast member. For example, in the movie, character A is a character played by actor C, so the speaker information can be the information of actor C. In this case, actor C is the target character.

[0083] In some embodiments, the speaker audio of some films is generated by post-dubbing, so the speaker audio is different from the actual voice of the cast and crew members playing the characters in the film. To this end, character recognition can also be performed based on the speaker audio to calculate the feature similarity between the preset human voice template and the speaker audio, thereby determining the speaker information.

[0084] S300: performing language conversion on the audio text according to a second language different from the first language to obtain a text to be converted.

[0085] Before performing language conversion on the audio text, it is necessary to determine the second language, so as to perform conversion on the audio text according to the second language, thereby obtaining the text to be converted, and the language information of the text to be converted is the second language. The second language information can be determined by the user. For example, if a user whose native language is Chinese watches an English film, in a scene where the user does not understand English, the user cannot understand the semantics of the lines based on the English lines spoken by the characters in the film. At this time, the first language of the audio text is English, and the user can select the second language as Chinese, so as to perform conversion on the audio text according to the Chinese language, thereby obtaining the text to be converted in Chinese, so that the user can understand the semantics of the lines.

[0086] S400: Generate a first audio according to the text to be converted, and retrieve a second audio according to the speaker information.

[0087] After obtaining the text to be converted, the text to be converted can be converted by format conversion according to the text to speech technology (Text To Speech, TTS) to generate a first audio. The first audio is the audio of the text to be converted spoken in a preset voice, and the language information of the first audio is the second language.

[0088] However, at this time, the first audio is only the same as the text to be converted in terms of text content, and does not match the speaker information in terms of sound features. Therefore, it is necessary to retrieve the second audio based on the speaker information. The second audio is an audio with the same timbre as the speaker information, which can be used as a sample audio for subsequent synthesis of the target audio.

[0089] In some embodiments, the second audio may be obtained from other videos based on the speaker information, for example, the audio of the speaker speaking in other videos or the audio of the speaker speaking in public may be obtained based on the speaker information. In order to improve the extraction accuracy of the second audio, after the second audio is extracted, background sound removal and noise reduction processing may be performed on the second audio to reduce the interference of noise and interfering sounds on the human voice in the second audio.

[0090] S500: Inputting the first audio and the second audio into an acoustic model, performing speech feature replacement according to the first audio feature and the second audio feature through the acoustic model, and outputting a target audio;

[0091] The acoustic model is obtained through a mask prediction training method. Mask prediction is used to characterize the method of completing the acoustic model training by masking at least part of the training data during the acoustic model training process and predicting the masked training samples. Through the mask prediction training method, the acoustic model's ability to understand the speech features in the first audio feature can be improved, thereby improving the efficiency and accuracy of subsequent speech feature replacement and improving the accuracy of target audio synthesis.

[0092] The first audio feature is an audio feature corresponding to the first audio. Since the first audio is an audio generated according to the text to be converted, the first audio feature is used to represent the voice feature of the speaker in the first audio. The second audio feature is an audio feature corresponding to the second audio, and is used to represent the voice feature of the target person in the second audio.

[0093] After obtaining the first audio and the second audio, the first audio and the second audio can be input into the acoustic model, and the acoustic model will extract the first audio feature of the first audio. After obtaining the first audio feature and the second audio feature, the acoustic model will perform voice feature replacement on the first audio feature and the second audio feature to replace the voice feature of the speaker in the first audio with the human voice feature of the target person, thereby generating a target audio in which the text to be converted is spoken by the human voice feature of the target person.

[0094] S600: Replace the audio to be converted with the target audio to obtain a target video.

[0095] After the target audio is obtained, the target audio replaces the audio to be converted, so that in the target video, the audio to be converted is switched from the first language to the second language according to the language information, thereby realizing the conversion of the audio of the lines in the video.

[0096] In some embodiments, the acoustic model includes an audio encoding module and an audio decoding module. Figure 2 As shown, the audio encoding module includes a feature encoding unit and a replacement unit. The feature encoding unit is configured to extract a first audio feature from a first audio, and to extract a second audio feature from a second audio. The replacement unit is configured to perform speech feature replacement on the first audio feature and the second audio feature according to a replacement network to obtain a target audio feature. The audio decoding module is configured to decode the target audio feature to obtain a target audio.

[0097] In the process of performing speech feature replacement in the replacement unit, such as Figure 3 As shown, the acoustic model can calculate the feature similarity value through the replacement unit, and the feature similarity value is the similarity value between the second audio feature and the first audio feature, and the replacement unit searches for the third audio feature in the second audio feature according to the feature similarity value based on the KNN algorithm, so as to replace the first audio feature with the third audio feature according to the replacement network to obtain the target audio feature. The third audio feature is a second audio feature whose feature similarity value is greater than or equal to the similarity threshold. The original first audio feature is replaced by the third audio feature similar to the first audio feature, so that the acoustic characteristics of the obtained target audio feature are more consistent with the acoustic characteristics of the target person, thereby improving the acoustic similarity of the target audio.

[0098] In some embodiments, the feature encoding unit extracts audio features from the audio by feature encoding. For example, the feature encoding unit performs feature encoding on the first audio and obtains the first audio feature as Then perform feature encoding on the second audio to obtain the second audio feature Where x∈R d is the feature representation of a frame, N s is the length of the first audio, N t is the length of the second audio.

[0099] In the process of calculating the feature similarity value, the replacement unit can generate a fourth audio feature by non-adjacent replacement according to the first audio feature, and the fourth audio feature is After generating the fourth audio feature, the replacement unit may use a proximity algorithm to calculate proximity values ​​between the second audio feature and the fourth audio feature, and generate a feature similarity value according to the proximity values, thereby determining the third audio feature in the second audio feature according to the feature similarity value, and the third audio feature is X t,e =X t ∪X e In the process of performing speech feature replacement, the replacement unit may replace the feature sequence X of the third audio feature. t,e Perform averaging to obtain the speech feature sequence to be replaced, and replace the feature sequence X in the first audio feature with the speech sequence to be replaced s , thus obtaining the final target audio encoding, namely

[0100] In some embodiments, if the audio encoding module wants to perform the above-mentioned audio synthesis effect of voice feature replacement, it is also necessary to perform a specific training process on the acoustic model. To this end, before inputting the first audio and the second audio into the acoustic model, it is also necessary to obtain a first training data set, the first training data set includes a preset number of first training data, the first training data is audio data for training the audio encoding module, and the first training data includes at least one speaker audio, so as to train the audio encoding module's ability to extract audio features.

[0101] Before training the audio coding module, it is necessary to initialize the parameters of the audio coding module, that is, to randomly initialize some parameters of the feature coding unit and the replacement unit. After the initialization is completed, the first training data can be input into the audio coding module to be trained to extract the first training feature of the first training data through the feature coding unit. Since the audio coding module needs to perform speech feature replacement between different audio features, during the training process, reference training data different from the first training data can be input into the audio coding module as sample audio data for speech feature replacement, so that speech feature replacement is performed through the first training feature and the reference training feature extracted according to the reference training data to complete the training process of the replacement unit.

[0102] In some embodiments, Figure 4As shown, two first training features from different first training data can also be input into the replacement unit to perform speech feature replacement on the two first training features to obtain the second training feature. After obtaining the second training feature, the acoustic model calculates the feature loss of the second training feature and the audio feature label through the loss function, wherein the audio feature label is used to characterize the standard audio feature after performing speech feature replacement, and is used to determine the convergence degree of the audio coding module. If the feature loss is less than or equal to the first loss threshold, it means that the feature accuracy of the second training feature output by the audio coding module after performing speech feature replacement reaches the accuracy of the audio feature label, and the acoustic model can output the audio coding module according to the current parameters of the feature coding unit and the current parameters of the replacement unit. If the feature loss is greater than the first loss threshold, it means that the feature accuracy of the second training feature output by the audio coding module after performing speech feature replacement does not reach the accuracy of the audio feature label. At this time, it is necessary to iteratively train the coding unit and the replacement unit through the first training data, and continuously update the parameters of the training coding unit and the parameters of the replacement unit during the training process until the calculated feature loss is less than or equal to the first loss threshold, and the training process of the audio coding module is completed.

[0103] In some embodiments, in order to improve the accuracy of the feature encoding unit in extracting audio features, the feature encoding unit may also extract audio features by mask prediction. Figure 5 As shown, the feature encoding unit includes a masking subunit, a first encoding unit and a second encoding unit. The masking subunit is configured to mask part of the audio features of the first training data, so as to facilitate subsequent feature prediction of the masked audio features. To this end, after the first training data is input into the feature encoding unit, the masking subunit can perform masking processing on the first training data to obtain masked training data. The masking subunit can perform masking on the first training data according to a preset masking ratio, and the masking ratio is used to characterize the proportion of masked data, and the masking ratio can be 30%-50%. For example, when the masking ratio is 50%, half of the data volume of the first training data is masked. Among them, the masked first training data can be a separate data portion, for example, the first 50% is masked, and the second 50% is not masked, so as to perform prediction on the first 50% of the audio features according to the second 50% of the first training data. The masked first training data may also be a plurality of data parts, for example, the first 25% of the first training data is masked, the second 25% of the first training data is not masked, the third 25% of the first training data is masked, and the fourth 25% of the first training data is not masked, so as to perform prediction on the masked data part according to the first training data not masked.

[0104] After completing the mask processing, the mask subunit can output the mask training data and input the mask training data to the first encoding unit. The first encoding unit is a CNN encoding unit, which can perform feature extraction on the mask training data with a convolution kernel of a preset size through a convolutional neural network and output mask audio features.

[0105] Since part of the training data in the masked training data is masked, the first encoding unit cannot extract features from the masked data. The first encoding unit can extract audio features from the data portion of the masked training data that is not masked to obtain masked audio features.

[0106] The first mask unit may input the masked audio feature into the second encoding unit, and the second encoding unit may be a Transformer encoding unit. The second encoding unit may further extract deep features from the masked audio feature, and perform feature prediction on the masked portion of the masked audio feature according to the extracted deep features to obtain the first training feature, thereby improving the accuracy of the audio feature extracted by the feature encoding unit according to the mask training method.

[0107] In the process of performing feature prediction by the second coding unit, it is necessary to perform prediction according to the attention weights of different feature sequences to improve the accuracy of feature prediction. To this end, the second coding unit also needs to calculate the attention weights of the feature sequences of the masked audio features before performing feature prediction.

[0108] Figure 6 This is a flow chart of calculating attention weights based on mask training features in an embodiment of the present application. Figure 6 , the mask training feature includes multiple mask feature sequences, the second encoding unit can obtain the sequence length of the mask audio feature, and generate a relative position index based on the sequence length, and the relative position index is the distance index between any two sequences in the mask audio feature. For example, for the mask feature sequence i, the second encoding unit can be based on the relative distance index r(i, j) between the mask feature sequence i and the mask feature sequence j. After generating the relative distance index, the second encoding unit can generate a relative position vector between the mask feature sequence i and the mask feature sequence j according to the distance index, and calculate the attention weight of the mask feature sequence i relative to the mask feature sequence j according to the relative position vector. It should be understood that in the above embodiment, it is the attention weight calculation process of the mask feature sequence i relative to the mask feature sequence j. In the actual training process, the attention weight can also be calculated based on any different mask feature sequences. The second encoding unit can perform feature prediction on the masked part of the mask audio feature according to the attention weights of multiple groups of mask feature sequences to obtain the first training feature.

[0109] The present application combines the relative position information of the mask feature sequence with the calculation of the attention weight in the process of calculating the attention weight, so as to achieve flexible control of the information interaction between different mask feature sequences, improve the feature correlation between different mask feature sequences, and thus improve the accuracy of the prediction.

[0110] In order to improve the accuracy of audio extraction, a clustering extraction method can also be used in the process of extracting training data, and the clustering extraction method can be jointly trained with the mask training process.

[0111] To this end, the second mask unit can also extract mask audio features by clustering extraction. To this end, the audio encoding module also includes a category mapping unit, which is configured to perform category mapping on the audio features. In the process of performing feature extraction on the mask training data by the second encoding unit, the second encoding unit can extract the mask features to be clustered of the mask training data. The mask features to be clustered are mask audio features that are not clustered. When the mask training data includes multiple speakers, the mask features to be clustered can include audio features of multiple speakers, but the audio features of other speakers will affect the extraction accuracy of the mask audio features.

[0112] To this end, the category mapping unit can perform category mapping on the mask features to be clustered according to the preset category code, thereby classifying the mask features to be clustered according to different speakers, reducing the influence of other speakers' audio on the extraction of mask audio features. The category mapping unit is composed of a mapping layer, which is used to perform category mapping on the mask features to be clustered according to the category code, thereby extracting mask audio features from the classified mask features to be clustered, and improving the extraction accuracy of mask audio features.

[0113] The audio coding module also includes a clustering module and a category coding unit, wherein the clustering module can provide the category coding to the audio coding module during the training process of the audio coding module. During the training process of the audio coding module, the category coding unit can obtain the category coding obtained by the clustering module after clustering the speech sample data, thereby providing the category coding to the category mapping unit. After obtaining the category coding, the category mapping unit can perform category mapping on the mask features to be clustered according to the category coding, and perform feature clustering on the mask features to be clustered according to the category coding, thereby obtaining the masked audio features.

[0114] It should be noted that the category coding unit does not participate in the feature extraction process of the audio coding module during the application process. It only introduces category coding into the audio coding module during the training process of the audio coding module to train the clustering extraction function of the audio coding module.

[0115] In the actual training process, in addition to training and updating the parameters of the feature coding unit, this embodiment also trains the audio coding module based on the category coding corresponding to the training data and the real category coding ID obtained by the clustering module to improve the accuracy of extracting audio features. To this end, the acoustic model also needs to perform training on the clustering module. Among them, the clustering module may include a feature extractor configured to extract audio features of the training data. In the process of training the clustering module, it is necessary to obtain a second training data set, and the second training data set includes a preset number of second training data, wherein the second training data can be audio data that is the same as or different from the first training data. The second training data can be based on LibriSpeech-960 and AISHELL-3 data, specifically, the speech sample data of 200 speakers are obtained respectively, and the number of clustering clusters is 200.

[0116] The second training data is input into the clustering module to be trained, the third training features of the second training data are extracted by the feature extractor, and cluster classification is performed on the third training features to generate a category code. During the cluster classification process, the feature extractor can perform feature similarity calculation on the third training features, and cluster the third training data whose feature similarity is less than or equal to the similarity threshold, that is, cluster similar audio features. After clustering, audio features of the same category can represent the audio features of an independent person, and the differences between audio features of different categories are relatively large, so that the audio features of each person can be extracted according to the category features to improve the feature extraction effect.

[0117] In order to improve the effect of clustering training, the clustering module may also include a first clustering module and a second clustering module, and the first clustering module and the second clustering module may respectively perform clustering training on the audio data of different characters in the second training data. The feature extractor of the first clustering module and the feature extractor of the second clustering module may be any two of the three feature extractors of softhubert, huber, and wav2vec2.0. For example, the feature extractor of the first clustering model adopts huber, and the feature extractor of the second clustering model adopts wav2vec2.0, etc.

[0118] To this end, the first clustering module and the second clustering module can also be trained simultaneously through the second training data, so that the first clustering module and the second clustering module can classify and process the audio data of different speakers in the second training data. Since the first clustering module and the second clustering module respectively use different feature extractors, the first clustering module and the second clustering module can respectively perform clustering from different dimensions, thereby improving the extraction method of different audio features, such as feature extraction in terms of timbre, rhythm, etc. Therefore, the combination of different feature extractions of the first clustering module and the second clustering module can make the training effect richer when the language conversion is subsequently performed.

[0119] In some embodiments, Figure 7 As shown, during the training process of the first clustering module and the second clustering module, the categories output by the first clustering module and the second clustering module need to be coded. For example, after the first clustering module clusters the second training data, different feature categories are obtained, which can be assigned ID1.1, ID1.2, ... ID1.9, etc. The corresponding second clustering model clusters the second training data to obtain different feature categories, which can be assigned ID2.1, ID2.2, ... ID2.9, etc. The purpose of category coding is to have a unique identifier for each category after clustering by the clustering module for distinction, so as to map and encode the categories in the subsequent language conversion training process.

[0120] In some embodiments, the acoustic model can also update the loss function of the audio coding module according to the first clustering module and the second clustering module. For example, the average cross entropy between the real category coding of the first clustering model and the predicted category coding of the audio coding module is minimized, and the average cross entropy between the real category coding of the second clustering model and the predicted category coding of the audio coding module is minimized, and the above loss is updated based on this, and the parameters of the audio coding module are updated at the same time. Through the above training method, the audio coding module's ability to classify timbre categories is further enhanced.

[0121] In order to facilitate the execution of the video conversion method based on mask prediction described in the above specification, some embodiments of the present application also provide a video conversion system based on mask prediction, including an audio extraction module, a text acquisition module, a conversion module, a speech generation module and an acoustic model;

[0122] The audio extraction module is configured to extract the audio to be converted of the video to be converted, wherein the audio to be converted includes at least one speaker audio;

[0123] The text acquisition module is configured to speech recognize the audio to be converted, obtain an audio text and speaker information corresponding to the speaker audio, and the language information of the audio text is the first language;

[0124] The conversion module is configured to perform language conversion on the audio text according to a second language different from the first language to obtain a text to be converted;

[0125] The speech generation module is configured to generate a first audio according to the text to be converted;

[0126] The acoustic model is configured to perform speech feature replacement according to a first audio feature and a second audio feature to output a target audio; the first audio feature is an audio feature corresponding to a first audio, and the second audio feature is an audio feature corresponding to a second audio; the target audio replaces the audio to be converted to obtain a target video, and the second audio is audio retrieved according to the speaker information; the acoustic model is obtained through a mask prediction training method.

[0127] It can be seen from the above technical scheme that the present application extracts the audio to be converted of the video to be converted, obtains the speaker information corresponding to the audio text of the first language and the speaker audio according to the audio to be converted, and performs language conversion on the audio text according to a second language different from the first language to obtain the text to be converted; generates a first audio according to the text to be converted, retrieves a second audio according to the speaker information, inputs the first audio and the second audio into the acoustic model, performs speech feature replacement according to the first audio feature and the second audio feature using the acoustic model obtained by training through mask prediction, outputs the target audio, replaces the audio to be converted with the target audio, and obtains the target video. The present application improves the accuracy of extracting speech features of the acoustic model by mask prediction, thereby improving the accuracy of extracting speech features of the first audio and the second audio, and improving the synthesis effect of the audio after speech conversion.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0129] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are intended to better explain the present disclosure, so that those skilled in the art can better use the embodiments.

Claims

1. A video conversion method based on mask prediction, characterized in that: include: Extracting audio to be converted from the video to be converted, wherein the audio to be converted includes at least one speaker audio; According to the audio to be converted, an audio text and speaker information corresponding to the speaker audio are obtained, wherein the language information of the audio text is a first language; Performing language conversion on the audio text according to a second language different from the first language to obtain a text to be converted; generating a first audio according to the text to be converted, and retrieving a second audio according to the speaker information; Inputting the first audio and the second audio into an acoustic model, so as to perform speech feature replacement according to the first audio feature and the second audio feature through the acoustic model, and output a target audio; the first audio feature is an audio feature corresponding to the first audio, and the second audio feature is an audio feature corresponding to the second audio; The acoustic model is obtained by a mask prediction training method; wherein the mask prediction is used to characterize mask processing of at least part of the training data, and to predict the masked training samples during the training process; The target audio replaces the audio to be converted to obtain the target video.

2. The video conversion method based on mask prediction according to claim 1, characterized in that: The acoustic model includes an audio encoding module and an audio decoding module, the audio encoding module includes a feature encoding unit and a replacement unit, and after the step of inputting the first audio and the second audio into the acoustic model, the method further includes: extracting a first audio feature of the first audio by the feature encoding unit, and extracting a second audio feature of the second audio by the feature encoding unit; The replacement unit calculates a feature similarity value, which is a similarity value between the second audio feature and the first audio feature, and extracts a third audio feature from the second audio feature according to the feature similarity value, and replaces the first audio feature according to the third audio feature to obtain a target audio feature; the third audio feature is a second audio feature whose feature similarity value is greater than or equal to a similarity threshold; The target audio feature is decoded by the audio decoding module to obtain the target audio.

3. The video conversion method based on mask prediction according to claim 2, characterized in that: The step of calculating the feature similarity value by the replacement unit comprises: Generate a fourth audio feature according to the first audio feature by the replacement unit, wherein the fourth audio feature is generated according to a non-contiguous replacement algorithm; Calculating a proximity value between the second audio feature and the fourth audio feature according to a proximity algorithm; A feature similarity value is generated based on the proximity value.

4. The video conversion method based on mask prediction according to claim 2, characterized in that: Before the step of inputting the first audio and the second audio into the acoustic model, the method further includes: Acquire a first training data set, where the first training data set includes a preset amount of first training data; Inputting the first training data into an audio encoding module to be trained, so as to extract a first training feature of the first training data through a feature encoding unit; Inputting at least two first training features into the replacement unit to perform speech feature replacement on the first training features to obtain second training features; Calculating a feature loss between the second training feature and an audio feature label, wherein the audio feature label is used to characterize a standard audio feature; If the feature loss is less than or equal to a first loss threshold, outputting the audio encoding module according to current parameters of the feature encoding unit and current parameters of the replacement unit; If the feature loss is greater than a first loss threshold, the feature encoding unit and the replacement unit are iteratively trained according to first training data.

5. The video conversion method based on mask prediction according to claim 4, characterized in that: The feature encoding unit includes a mask subunit, a first encoding unit, and a second encoding unit. The extracting the first training feature of the first training data by the feature encoding unit includes: Inputting the first training data into the feature encoding unit so as to perform mask processing on the first training data through the mask subunit to obtain masked training data; Performing feature extraction on the mask training data by the first encoding unit to obtain mask audio features; The second encoding unit performs feature prediction on the masked part of the masked audio feature to obtain a first training feature.

6. The video conversion method based on mask prediction according to claim 5, characterized in that: The step of performing feature prediction on the masked part of the masked audio feature by the second encoding unit comprises: Obtaining a sequence length of the masked audio feature, and generating a relative position index according to the sequence length, wherein the relative position index is a distance index between two sequences in the masked audio feature; Generate a relative position vector according to the distance index; An attention weight is calculated for the relative position vector, and a feature prediction is performed on the masked portion of the masked audio feature according to the attention weight to obtain a first training feature.

7. The video conversion method based on mask prediction according to claim 5, characterized in that: The audio encoding module further includes a category mapping unit, and the step of performing feature extraction on the mask training data by the second encoding unit includes: Extracting, by the second encoding unit, mask features to be clustered of the mask training data; The category mapping unit performs category mapping on the mask features to be clustered to obtain the mask audio features.

8. The video conversion method based on mask prediction according to claim 7, characterized in that: The audio encoding module further includes a clustering module and a category encoding unit, and the step of performing category mapping on the mask features to be clustered includes: Acquiring a category code through the category coding unit, wherein the category code is obtained by clustering the speech sample data by the clustering module; The category mapping unit performs category mapping on the mask features to be clustered according to the category codes to obtain the mask audio features.

9. The video conversion method based on mask prediction according to claim 8, characterized in that: The clustering module includes a feature extractor. Before the step of obtaining the category code by the category coding unit, the method further includes: Acquire a second training data set, where the second training data set includes a preset amount of second training data; Inputting the second training data into a clustering module to be trained, so as to extract a third training feature of the second training data through the feature extractor; Cluster classification is performed on the third training feature to generate the category code.

10. A video conversion system based on mask prediction, characterized in that: It includes audio extraction module, text acquisition module, conversion module, speech generation module and acoustic model; The audio extraction module is configured to extract the audio to be converted of the video to be converted, wherein the audio to be converted includes at least one speaker audio; The text acquisition module is configured to obtain an audio text and speaker information corresponding to the speaker audio according to the audio to be converted, and the language information of the audio text is the first language; The conversion module is configured to perform language conversion on the audio text according to a second language different from the first language to obtain a text to be converted; The speech generation module is configured to generate a first audio according to the text to be converted; The acoustic model is configured to perform speech feature replacement according to a first audio feature and a second audio feature, and output a target audio; the first audio feature is an audio feature corresponding to the first audio, and the second audio feature is an audio feature corresponding to the second audio; the target audio replaces the audio to be converted to obtain a target video, and the second audio is the audio retrieved according to the speaker information; wherein the mask prediction is used to characterize mask processing of at least part of the training data, and to predict the training samples after mask processing during the training process; The acoustic model is obtained through a mask prediction training method.

Images