Video conversion method and system based on feature clustering

Abstract

The application provides a video conversion method and system based on feature clustering. The method extracts the to-be-converted audio of the to-be-converted video, and according to the audio text of the to-be-converted audio, the language information of the audio text is the first language. The language conversion is performed on the audio text according to the second language to obtain the to-be-converted text. The to-be-converted text and the reference audio are input into an acoustic model to perform voice prediction according to the text features and the reference audio through the acoustic model, output the target audio, replace the to-be-converted audio with the target audio, and obtain the target video. Through voice clustering extraction, in the case of low samples, the audio features of the reference audio with higher precision are extracted, so that the audio precision of the target audio is improved when the target audio is synthesized, and the conversion effect of voice translation is improved.

Description

[0001] This application claims priority to Chinese patent application No. 202311479579.X, filed on November 8, 2023, with the State Intellectual Property Office of China. The entire contents of the latter are incorporated herein by reference. Technical Field

[0002] This application relates to the field of speech conversion technology, and in particular to a video conversion method and system based on feature clustering. Background Technology

[0003] When watching films from different countries, users cannot directly understand the meaning of the dialogue from the audio alone because the film's language differs from their own. Therefore, users need to continuously read subtitles while watching the film to comprehend the meaning of the characters' lines.

[0004] To improve the viewing experience, the dialogue can be translated into other languages ​​before being dubbed in post-production, allowing users to intuitively understand the meaning of the lines based on the dubbing. However, this post-dubbing method is time-consuming and needs to be completed before the video plays, making it impossible to translate and play the dialogue in real time.

[0005] Therefore, speech-to-text conversion technology can be used to convert the language of a film into the user's desired language while preserving the vocal characteristics of the characters, thus achieving real-time translation. To achieve this, the speech-to-text conversion model needs to be trained with a large number of voice samples from specific individuals. However, it is difficult to obtain a large number of voice samples for a particular individual, resulting in low accuracy of speech-to-text conversion in scenarios with few samples. Summary of the Invention

[0006] To address the issue of low speech conversion accuracy when translating spoken language in scenarios with low sample counts, some embodiments of this application provide a video conversion method and system based on feature clustering.

[0007] In a first aspect, some embodiments of this application provide a video conversion method based on feature clustering, including:

[0008] Extract the audio from the video to be converted;

[0009] Based on the audio to be converted, an audio text is obtained, wherein the language information of the audio text is the first language;

[0010] The audio text is converted to a second language, which is different from the first language, to obtain the text to be converted.

[0011] The text to be converted and the reference audio are input into an acoustic model to perform speech prediction based on the text features and the reference audio features, and output the target audio. The reference audio is audio data from the same sound source as the audio to be converted. The text features are text features extracted from the text to be converted, and the reference audio features are audio features extracted based on the reference audio. The acoustic model is trained using feature clustering. Feature clustering refers to feature processing on training samples.

[0012] Replace the audio to be converted with the target audio to obtain the target video.

[0013] In some embodiments, the acoustic model includes an audio encoding module, a text embedding module, an audio prediction module, and an audio decoding module. The method for inputting the text to be converted and the reference audio into the acoustic model includes:

[0014] The audio encoding module performs speech clustering extraction on the reference audio to obtain reference audio features, and the text embedding module extracts text features of the text to be converted.

[0015] The reference audio features and the text features are input into the audio prediction module to obtain the target audio code;

[0016] The target audio is obtained by decoding the target audio code using the audio decoding module.

[0017] In some embodiments, the audio prediction module includes a quantization unit, an autoregressive unit, and a non-autoregressive unit. The step of inputting the reference audio features and the text features into the audio prediction module includes:

[0018] The reference audio feature is quantized a preset number of times by the quantization unit to obtain a first audio vector and a second audio vector. The first audio vector is the audio vector obtained by performing quantization on the reference audio feature for the first time, and the second audio vector is the audio vector obtained by performing quantization on the reference audio feature other than the first audio vector.

[0019] The first audio vector and the text features are input into the autoregressive unit to perform speech prediction on the first audio vector based on the text features, and obtain a first prediction result;

[0020] The second audio vector is input into the non-autoregressive unit to perform speech prediction on the second audio vector and obtain a second prediction result;

[0021] By concatenating the first prediction result and the second prediction result, the target audio code is obtained.

[0022] In some embodiments, the quantization unit includes a hierarchical structure formed by a plurality of quantization subunits, wherein the input of a quantization subunit is the output of the previous quantization subunit, and the output of a quantization subunit is the input of the next quantization subunit. Performing a quantization operation on the reference audio feature through the quantization unit includes:

[0023] The reference audio features are input into the hierarchical structure to perform the first quantization operation on the reference audio features through the first quantization subunit to obtain the first audio vector;

[0024] The first audio vector is input to the second quantization subunit, and residual vector quantization is performed on the first audio vector through the quantization subunit to obtain the second audio vector.

[0025] In some embodiments, the audio prediction module further includes a style extraction unit configured to extract style features from the audio vector, and the step of inputting the second audio vector into the non-autoregressive unit includes:

[0026] The second audio vector is input into the style extraction unit to generate a style embedding matrix, which is used to characterize the vectors corresponding to different style features in the second audio vector.

[0027] The attention weights of the style features are calculated using an attention mechanism;

[0028] A style feature sequence is generated based on the attention weights, and the style feature sequence is input into the non-autoregressive unit.

[0029] In some embodiments, the audio encoding module includes an audio encoder, which includes a feature encoding unit and a category mapping unit. The step of performing speech clustering extraction on the reference audio through the audio encoding module includes:

[0030] The clustering features of the reference audio are extracted using the feature encoding unit.

[0031] The reference audio features are obtained by performing category mapping on the features to be clustered according to the preset category code by the category mapping unit.

[0032] In some embodiments, the method further includes:

[0033] Obtain a first training dataset, which includes a preset number of first training data.

[0034] The first training data is input into the audio encoder to be trained, so as to extract the first training feature of the first training data through the feature encoding unit.

[0035] The first training feature is subjected to category mapping by the category mapping unit to obtain the second training feature;

[0036] Calculate the feature loss between the second training feature and the audio feature label, where the audio feature label is used to characterize standard audio features;

[0037] If the feature loss is less than or equal to the first loss threshold, the audio encoder is output according to the current parameters of the feature encoding unit and the current parameters of the category mapping unit.

[0038] If the feature loss is greater than the first loss threshold, then the feature encoding unit and the category mapping unit are iteratively trained based on the first training data.

[0039] In some embodiments, the audio encoder further includes a clustering module and a category encoding unit, and the step of performing category mapping on the first training features through the category mapping unit includes:

[0040] The category code is obtained through the category coding unit, and the category code is obtained by the clustering module after performing clustering based on the speech sample data.

[0041] The category mapping unit performs category mapping on the first training feature according to the category encoding to obtain the second training feature.

[0042] In some embodiments, the clustering module includes a feature extractor, and before the step of obtaining the category code through the category coding unit, the method further includes:

[0043] Obtain a second training dataset, which includes a preset number of second training data.

[0044] The second training data is input into the clustering module to be trained, so that the third training feature of the second training data can be extracted by the feature extractor;

[0045] Clustering classification is performed on the third training feature to generate the category code.

[0046] Secondly, this application provides a video conversion system based on feature clustering, including an audio extraction module, a text acquisition module, a conversion module, and an acoustic model;

[0047] The audio extraction module is configured to extract the audio to be converted from the video to be converted;

[0048] The text acquisition module is configured to recognize the audio to be converted by speech to obtain audio text, wherein the language information of the audio text is a first language;

[0049] The conversion module is configured to convert the audio text to a second language different from the first language to obtain the text to be converted.

[0050] The acoustic model is configured to perform speech prediction based on text features and reference audio features, and output target audio; the reference audio is audio data from the same sound source as the audio to be converted, the text features are text features extracted from the text to be converted, and the reference audio features are audio features extracted by performing speech clustering based on the reference audio; the target audio is concatenated to the audio to be converted to obtain the target video; wherein, feature clustering refers to feature processing on training samples;

[0051] The acoustic model is obtained through training by feature clustering.

[0052] As can be seen from the above technical solutions, this application provides a video conversion method and system based on feature clustering. The method extracts the audio to be converted from the video to be converted; based on the audio text of the audio text, the language information of the audio text is a first language; language conversion is performed on the audio text according to a second language to obtain the text to be converted; the text to be converted and a reference audio are input into an acoustic model, which performs speech prediction based on the text features and the reference audio, outputs the target audio, and replaces the audio to be converted with the target audio to obtain the target video. This application, through speech clustering extraction, extracts higher-precision audio features from the reference audio even with low sample sizes, thereby improving the audio accuracy of the target audio during synthesis and enhancing the speech conversion effect. Attached Figure Description

[0053] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart illustrating a video conversion method based on feature clustering provided in some embodiments of this application;

[0055] Figure 2 Model structure diagrams of acoustic models provided in some embodiments of this application;

[0056] Figure 3 This is a flowchart illustrating the generation of style feature sequences based on style extraction units in some embodiments of this application;

[0057] Figure 4 This is a flowchart illustrating the training of an audio encoder in some embodiments of this application;

[0058] Figure 5 This document provides a flowchart illustrating the introduction of category coding during the training of the audio encoder in some embodiments of this application.

[0059] Figure 6 This is a flowchart of the training clustering module in some embodiments of this application. Detailed Implementation

[0060] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0061] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0062] The terms "first," "second," "third," etc., used in the specification and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms can be used interchangeably where appropriate.

[0063] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0064] Users can play videos through smart devices, such as movies, TV series, operas, or variety shows. When watching films from different countries, because the film's language is different from the user's, the user cannot directly understand the meaning of the dialogue based on the characters' words. For example, when a Chinese user watches an English film, if the user cannot understand English, they cannot directly know the meaning of the dialogue based on the audio of the characters' words.

[0065] Therefore, some video resources can include subtitle resources, that is, displaying subtitles in the corresponding language on the screen while the video is playing. For example, when playing a video in English, Chinese subtitles can be added on the screen so that Chinese users can understand the meaning of the dialogue. This requires users to watch the video and subtitles simultaneously to understand the meaning of the characters' dialogue. Although the subtitles are on the screen, repeatedly watching content in different positions affects the user's viewing experience.

[0066] To improve the viewing experience, the dialogue can be translated into other languages ​​before being dubbed in post-production, allowing users to intuitively understand the meaning of the lines based on the dubbing. However, this post-dubbing method is time-consuming and needs to be completed before the video plays, making it impossible to translate and play the dialogue in real time.

[0067] Therefore, speech-to-text conversion technology can be used to convert the language of a film into the user's desired language while preserving the vocal characteristics of the characters, thus achieving real-time translation. To achieve this, the speech-to-text conversion model needs to be trained with a large number of voice samples from specific individuals. However, it is difficult to obtain a large number of voice samples for a particular individual, resulting in low accuracy of speech-to-text conversion in scenarios with few samples.

[0068] To address the issue of low speech conversion accuracy when translating spoken language in scenarios with low sample counts, some embodiments of this application provide a video conversion method based on feature clustering, which can be applied to electronic devices capable of playing video media assets. Figure 1 A flowchart illustrating a video conversion method based on feature clustering provided in this application embodiment. See also... Figure 1 The method includes:

[0069] S100: Extract the audio from the video to be converted.

[0070] The video to be converted is the video for which language conversion will be performed. The video should include audio dialogue, which can include both character audio and narration audio. For example, if character A in the video says, "The weather is so nice today," then "The weather is so nice today" is the character audio. Another example is playing audio generated from the video's narration using a preset character's voice, such as "Character A is taking a walk in the nice weather," then "Character A is taking a walk in the nice weather" is the narration audio.

[0071] The audio to be converted can be the character audio and the narration audio in the video to be converted. When there is a scene where multiple characters are speaking simultaneously in the audio to be converted, character recognition can be performed based on the current video frame, and selection can be made according to the recognized characters to determine the target character for audio conversion, and the character audio of the target character can be obtained, that is, the audio to be converted.

[0072] If it is necessary to convert the language of the character audio corresponding to all characters in the current video frame, the character audio set corresponding to each character is determined according to each character, and after obtaining the character audio set of each character, subsequent language conversion is performed based on the character audio in each character audio set to improve the accuracy when different characters speak alternately.

[0073] S200: Obtain the audio text according to the audio to be converted.

[0074] After obtaining the audio to be converted, the audio to be converted can be input into a speech recognition model to perform speech recognition on the audio to be converted through the speech recognition model, and the audio text corresponding to the audio to be converted can be obtained, so as to facilitate subsequent language conversion of the audio text.

[0075] In some embodiments, it is also necessary to determine the language information of the audio text before performing speech conversion on the corresponding audio text. For this purpose, after the audio text is obtained by speech recognition, language detection can be performed on the audio text to obtain the language information of the audio text. Among them, the language information of the audio text is the first language. For example, when the audio text is "Hello", it can be detected that "Hello" is an English text, so as to determine that the first language is English.

[0076] It should be understood that the Chinese language and the English language in the above embodiments are only illustrative. In the actual application process, conversion between other different languages can also be achieved. Or, conversion between different characteristic dialects under the same language can also be achieved, that is, the audio text remains unchanged, and the audio to be converted is converted through different dialects.

[0077] In some embodiments, in addition to speech recognition, the audio text can also be obtained through other means. For example, when playing the video to be converted, the subtitle data of the video to be converted can be obtained, and text recognition can be performed on the subtitles to obtain the audio text corresponding to the audio to be converted. Another example is that the text resource track of the video to be converted can be obtained, and the audio text can be directly obtained through the text resource track. Among them, the video to be converted can include multiple tracks, such as an audio resource track, a video resource track, etc. The text resource track is the text content output by each character or the narration in the video to be converted. Therefore, the video conversion device can obtain the audio text according to the text resource track.

[0078] S300: The audio text is converted to a second language different from the first language to obtain the text to be converted.

[0079] Before performing language conversion on audio text, a second language needs to be determined. The audio text is then converted based on this second language to obtain the text to be converted. This second language information is the language information of the text to be converted. This second language information can be determined by the user. For example, if a user whose native language is Chinese is watching an English movie, and they don't understand English, they cannot understand the meaning of the dialogue based on the characters' spoken English lines. In this case, the first language of the audio text is English, and the user can select Chinese as the second language. The audio text is then converted based on this Chinese language to obtain the text to be converted in Chinese, allowing the user to understand the meaning of the dialogue.

[0080] S400: Input the text to be converted and the reference audio into the acoustic model, so that the acoustic model can perform speech prediction based on the text features and the reference audio features, and output the target audio.

[0081] After obtaining the text to be converted, it is necessary to generate the converted target audio based on the text. For this purpose, an audio sample is used as a reference, and its audio features are incorporated into the target audio generation process. For ease of description, this application refers to the reference audio sample as reference audio, which is audio data from the same sound source as the audio to be converted.

[0082] In some embodiments, the method for obtaining reference audio may include obtaining speaker information of the audio to be converted. For example, if the speaker of the audio to be converted is person A, i.e., person A is the target person, then speaker information can be generated based on person A, and reference audio can be retrieved based on the speaker information as a search criterion. For example, audio data of person A in other movies can be retrieved based on the speaker information, or video data including person A speaking can be retrieved, and the audio data of person A speaking in the video data can be extracted to obtain the reference audio.

[0083] After obtaining the reference audio, the text to be converted and the reference audio can be input into an acoustic model to extract text features from the text to be converted and reference audio features from the reference audio. Speech prediction is then performed based on the text features and reference audio features to generate the target audio. The target audio is the audio data of the text to be converted spoken by a voice from the same sound source as the reference audio. Since the language information of the text to be converted is a second language, the target audio is the language-converted audio to be converted.

[0084] In this embodiment, the acoustic model is obtained through feature clustering training. Feature clustering refers to performing feature processing on the training data, thereby enabling the acoustic model trained based on the training data to have the ability to cluster features. This can improve the accuracy of feature extraction during the process of extracting text features of the text to be converted, as well as during the process of extracting reference audio features, thereby improving the accuracy of the target audio.

[0085] S500: Replace the audio to be converted with the target audio to obtain the target video.

[0086] After obtaining the target audio, the audio to be converted is replaced with the target audio, so that in the target video, the audio of the characters is switched from the audio to be converted with the language information of the first language to the target audio with the language information of the second language, thus realizing the conversion of video dialogue.

[0087] In some embodiments, the acoustic model includes an audio encoding module, a text embedding module, an audio prediction module, and an audio decoding module. The audio encoding module is configured to perform speech clustering extraction on a reference audio, and the text embedding module is configured to extract text features from the text to be converted.

[0088] After inputting the reference audio and the text to be converted into the acoustic model, the reference audio is input to the audio encoding module, and the reference audio features are output from the audio encoding module. The text to be converted is input to the text embedding module, and the text features are output from the text embedding module. After obtaining the reference audio and text features, they can be input to the audio prediction module, which performs speech prediction based on these features to obtain the target audio encoding. For this purpose, the outputs of the audio encoding module and the text embedding module are connected to the input of the audio prediction module, respectively, to facilitate the input of the reference audio and text features. After the audio prediction module outputs the target audio encoding, the audio decoding module decodes the target audio encoding to obtain the target audio.

[0089] In some embodiments, such as Figure 2 As shown, the audio prediction module includes a quantization unit, an autoregressive unit, and a non-autoregressive unit. The quantization unit is configured to perform a preset number of quantization operations on the reference audio features to obtain a first audio vector and a second audio vector. The first audio vector is the audio vector obtained by performing the first quantization operation on the reference audio features, and the second audio vector is the audio vector obtained by performing quantization operations on the reference audio features other than the first audio vector.

[0090] The quantization unit can be a Residual Vector Quantizer (RVQ). An RVQ can comprise a hierarchical structure of several Vector Quantizers (VQs). The input to a VQ is the output of the previous VQ, and the output of the previous VQ is the input to the next VQ. Taking an RVQ with eight VQ layers as an example, after inputting the reference audio feature into the hierarchical structure, the first VQ layer receives the reference audio feature as input and performs residual quantization on it to output a first audio vector. This first audio vector is then input to the second VQ layer, which performs residual quantization on the first VQ and inputs the resulting residual vector to the third VQ layer in hierarchical order. This process continues until all VQ layers have completed the quantization process, resulting in eight residual vectors. The first residual vector is the first audio vector, and the remaining residual vectors, from the second to the eighth VQ, are the second audio vectors.

[0091] After obtaining the first and second audio vectors, the first audio vector and text features can be input into an autoregressive model (AR). The AR can then perform speech prediction on the first audio vector using the text features, obtaining a first prediction result. The AR needs to perform the speech prediction process based on the feature weights of the first audio vector. To achieve this, the AR can employ a causal attention mechanism, which relies on the previous features of the feature sequence of the first audio vector to calculate the attention weight of the current feature, thereby completing the prediction process for the first audio vector.

[0092] Since the acoustic characteristics represented by the initial stage of residual vector quantization (i.e., the first audio vector output by the first quantization subunit) focus on speaker information, the autoregressive unit can perform speech prediction based on the acoustic characteristics of the target person in the first audio vector. Furthermore, combining text features ensures that the length of the first prediction result matches the length of the text to be converted, avoiding discontinuities in the target audio encoding caused by a mismatch between the lengths of the first prediction result and the text to be converted, thus improving the accuracy of the first prediction result.

[0093] In some embodiments, a second audio vector can be input into a non-autogressive (NAR) unit to perform speech prediction on the second audio vector, obtaining a second prediction result. The NAR unit needs to perform the speech prediction process based on the feature weights of the second audio vector. Therefore, the NAR unit can employ a non-causal attention mechanism, meaning that in calculating the attention weights, it does not rely on the previous features of the feature sequence of the second audio vector, but instead calculates the attention weights based on the entire feature sequence, thereby completing the prediction process for the second audio vector.

[0094] Because the acoustic characteristics characterized by the subsequent stages of residual vector quantization (i.e., the second audio vector output by the quantization subunits from layers 2 to 8) focus on acoustic information that has a predictive impact on the target audio, such as pitch, timbre, and intensity, the non-autoregressive unit can generate prediction results that are more similar to the acoustic information of the target person during the prediction of the second audio vector, thus improving the language conversion effect of the audio to be converted. Furthermore, due to the non-causal attention mechanism employed by the non-autoregressive unit, parallel prediction can be achieved. Therefore, during the prediction of the second audio vector, a second prediction result can be quickly generated for use in the generation of the target audio. Both the autoregressive and non-autoregressive units adopt a transformer decoder architecture.

[0095] After obtaining the first and second prediction results, the acoustic model can concatenate the first and second prediction results to combine the prediction results of speaker information and acoustic information to generate the target audio code. The target audio code is then decoded by the audio decoding module to obtain the target audio.

[0096] During acoustic model training, the output of the residual vector quantization unit, based on the different quantization subunits corresponding to the training audio samples, can be represented as (c * :,1 ,c * :,2 ,...,c * :,8 Based on this, autoregressive and non-autoregressive units are trained using training text, and then the autoregressive and non-autoregressive units generate the predicted acoustic feature codes (c :,1 ,c :,2 ,...,c :,8 The training optimization objective is maxp(C|x,C). * ).

[0097] The loss function corresponding to the autoregressive unit can be expressed as:

[0098]

[0099] θAR C represents the hyperparameters of the autoregressive unit; C represents the output representation of the quantization subunit. :,1 This is the first audio vector.

[0100] The loss function corresponding to a non-autoregressive unit can be expressed as:

[0101]

[0102] θ NAR+content For the hyperparameters of the non-autoregressive unit; c :,2:8 This is the second audio vector.

[0103] The entire feature training and prediction sequence can be represented as:

[0104]

[0105] In some embodiments, to make the target audio more consistent with the acoustic information of the target person, the audio prediction module may further include a style extraction unit. The style extraction unit is configured to extract style features from the audio vector. For this purpose, the style extraction unit can be located between the quantization unit and the autoregressive and non-autoregressive units; that is, the output of the quantization unit is connected to the input of the style unit, and the output of the style unit is connected to the outputs of both the autoregressive and non-autoregressive units. Taking the extraction of style features from a second audio vector by the style extraction unit as an example, it should be noted that the process of extracting the first audio vector by the style extraction unit is the same as the process of extracting the second audio vector. Therefore, the style feature extraction process described later can be referred to concurrently.

[0106] After the quantization unit quantizes the reference audio features, the acoustic model can input the second audio vector into the style extraction unit. The style extraction unit can traverse different style features in the second audio vector, such as pitch, timbre, intonation, speech rate, and volume. Style features can also include emotional features to represent the target speaker's emotional state, such as happiness, sadness, or anger, so as to generate target audio that better matches the target speaker's speaking style. The style extraction unit generates a style embedding matrix based on different style features, which represents the vectors corresponding to different style features in the second audio vector.

[0107] The style extraction module can calculate the attention weights of different style features in the style embedding matrix based on the attention mechanism to obtain the importance of different style features to the overall second audio vector. Based on the attention weights, a style feature sequence is generated, which retains the attention distribution of different style features. This allows the attention weights of style features to be incorporated into the subsequent speech prediction process, ensuring that the predicted speech style matches the target speaker's style. Finally, the style feature sequence is input into a non-autoregressive unit to perform speech prediction. It should be noted that the style feature sequence has the same structure as the second audio vector, only incorporating style features and their corresponding attention weights. Therefore, the non-autoregressive unit can perform speech prediction on the style feature sequence according to the prediction process for the second audio vector. The first audio vector can also have different style features extracted by the style extraction unit to generate a corresponding style feature sequence. This style feature sequence, along with the text features, is then input into an autoregressive unit to perform the speech prediction process.

[0108] In some embodiments, such as Figure 3 As shown, the style extraction module can incorporate style control parameters during attention weight calculation. These parameters determine the emotional tone of the current audio based on the video content before and after the current frame, or allow for user selection, such as providing an emotional style selection on the front-end interface, each corresponding to a different style control parameter. The style extraction module can assign a style control parameter to each style feature, with values ​​ranging from 0.5 to 2. A higher style control parameter indicates that the style feature should be strengthened, while a lower parameter indicates that the style feature should be weakened. The style extraction module can multiply the style control parameters by the style embedding matrix before calculating attention weights, generating a style feature sequence that significantly improves the style in the final audio presentation.

[0109] In some embodiments, the audio encoding module further includes an audio encoder for extracting reference audio features from the reference audio. To improve the accuracy of the extracted audio features, the audio encoder can perform speech clustering extraction on the reference speech, that is, cluster the audio of different speakers in the reference audio to distinguish between different speakers' audio. After performing clustering, reference audio features are extracted from the reference audio. For this purpose, the audio encoder may include a feature encoding unit and a category mapping unit. The feature encoding unit consists of an embedding layer for extracting the features to be clustered in the reference audio, which are the audio features that have not been clustered. When the reference audio includes multiple speakers, the features to be clustered may include the audio features of multiple speakers, including the reference audio features, which are the audio features of the target speaker in the reference audio. However, the audio features of other speakers may affect the extraction accuracy of the reference audio features.

[0110] To address this, the category mapping unit can perform category mapping on the features to be clustered based on a preset category code, thereby classifying the features according to different speakers and reducing the influence of other speakers' audio on the extraction of reference audio features. The category mapping unit consists of a mapping layer used to perform category mapping based on the category code, thereby extracting the reference audio features from the features to be clustered and improving the extraction accuracy of the reference audio features.

[0111] In some embodiments, to achieve the aforementioned speech clustering extraction effect, the audio encoder also needs to perform a specific training process. During training, the feature encoding unit and the category mapping unit are first initialized, that is, some parameters of the feature encoding unit and the category mapping unit are randomly initialized. After initialization, a first training dataset is obtained, which includes a preset number of first training data points used to train the audio encoder to be trained. After inputting the first training data into the audio encoder to be trained, the feature encoding unit can extract the first training features from the first training data. The first training features correspond to the clustering features of the acoustic model in the application stage. Therefore, the acoustic model performs category mapping on the first training features through the category mapping unit to obtain second training features, which correspond to the reference audio features of the acoustic model in the application stage.

[0112] After obtaining the second training feature, the feature loss between the second training feature and the audio feature label can be calculated using a loss function. The audio feature label is used to represent the standard audio feature, and the standard audio feature is used as a reference to determine the output of the second training feature. The loss function can be expressed as follows:

[0113]

[0114] Where i is the id of the cluster category, e i Let be the embedding vector of the trainable audio encoder, sim be the cosine similarity, and τ be the hyperparameter.

[0115] like Figure 4 As shown, when the feature loss is less than or equal to the first loss threshold, it indicates that the second training features extracted by the audio encoder meet the output standard, and the audio encoder completes the training process. At this point, the audio encoder can be output based on the current parameters of the feature encoding unit and the current parameters of the category mapping unit to complete the training. If the feature loss is greater than the first loss threshold, it indicates that the accuracy of the second training features extracted by the audio encoder has not reached the accuracy of the audio feature labels. Therefore, iterative training of the audio encoder is required, that is, iterative training of the feature encoding unit and the category mapping unit using the first training data.

[0116] In some embodiments, such as Figure 5 As shown, the audio encoder also includes a clustering module and a category encoding unit. The clustering module provides category codes to the audio encoder during training. During training, the category encoding unit obtains the category codes obtained by the clustering module after performing clustering on the speech sample data, and then provides these category codes to the category mapping unit. After obtaining the category codes, the category mapping unit performs category mapping on the first training features according to the category codes, clustering the first training features according to the category codes to obtain the second training features.

[0117] It should be noted that the category coding unit does not participate in the feature extraction process of the audio encoder during application. It is only introduced into the audio encoder during the training process to train the clustering extraction function of the audio encoder.

[0118] In the actual training process, in addition to training and updating the parameters of the feature encoding unit, this embodiment also trains the audio encoder based on the category encoding corresponding to the training data and the real category encoding ID obtained by the clustering module to improve the accuracy of audio feature extraction. For this purpose, the acoustic model also needs to train the clustering module. The clustering module may include a feature extractor configured to extract audio features from the training data. During the training of the clustering module, a second training dataset needs to be obtained. The second training dataset includes a preset number of second training data points, which may be the same as or different from the first training data. The second training data may be based on LibriSpeech-960 and AISHELL-3 data; specifically, it obtains speech sample data from 200 speakers, with 200 clusters.

[0119] The second training data is input into the clustering module to be trained. A feature extractor extracts the third training features from the second training data and performs clustering classification on these third training features to generate category codes. During the clustering classification process, the feature extractor can calculate feature similarity on the third training features and cluster the third training data whose feature similarity is less than or equal to a similarity threshold, i.e., clustering similar audio features. After clustering, audio features of the same category can represent the audio features of an independent person, while audio features of different categories differ significantly. This facilitates the extraction of audio features for each person based on category features, thereby improving the feature extraction effect.

[0120] To improve the clustering training effect, the clustering module may further include a first clustering module and a second clustering module. The first and second clustering modules can respectively perform clustering training on the audio data of different individuals in the second training data. The feature extractors of the first and second clustering modules can be any two of three types of feature extractors: SoftHubert, Hubert, and WAV2VEC2.0. For example, the first clustering model can use Hubert as its feature extractor, and the second clustering model can use WAV2VEC2.0 as its feature extractor.

[0121] Therefore, the first and second clustering modules can be trained simultaneously using the second training data, enabling them to classify audio data from different speakers within the second training data. Since the first and second clustering modules employ different feature extractors, they can cluster from different dimensions, improving the extraction of various audio features, such as timbre and prosody. Thus, the combination of different feature extraction methods from the first and second clustering modules can enrich the training results when performing language transformations subsequently.

[0122] In some embodiments, such as Figure 6 As shown, during the training process of the first and second clustering modules, it is necessary to encode the categories output by the first and second clustering modules. For example, after the first clustering module clusters the second training data, it obtains different feature categories, which can be assigned ID1.1, ID1.2, ..., ID1.9, etc. Similarly, after the second clustering model clusters the second training data, it obtains different feature categories, which can be assigned ID2.1, ID2.2, ..., ID2.9, etc. The purpose of category encoding is to ensure that each category after clustering by the clustering module has a unique identifier for differentiation, so as to facilitate category mapping and encoding during subsequent language transformation training.

[0123] In some embodiments, the acoustic model can also update the loss function of the audio encoder based on the first clustering module and the second clustering module. For example, the average cross-entropy between the true class code of the first clustering model and the predicted class code of the audio encoding module is minimized, while the average cross-entropy between the true class code of the second clustering model and the predicted class code of the audio encoding module is also minimized. Based on this, the loss function is updated, and the parameters of the audio encoding module are updated. Through the above training method, the audio encoder's ability to classify timbre categories is further enhanced.

[0124] To facilitate the execution of the feature clustering-based video conversion method described above, some embodiments of this application also provide a feature clustering-based video conversion system, including an audio extraction module, a text acquisition module, a conversion module, and an acoustic model;

[0125] The audio extraction module is configured to extract the audio to be converted from the video to be converted;

[0126] The text acquisition module is configured to obtain audio text based on the audio to be converted, wherein the language information of the audio text is a first language;

[0127] The conversion module is configured to convert the audio text to a second language different from the first language to obtain the text to be converted.

[0128] The acoustic model is configured to perform speech prediction based on text features and reference audio features, and output target audio; the reference audio is audio data from the same sound source as the audio to be converted, the text features are text features extracted from the text to be converted, and the reference audio features are audio features extracted based on the reference audio; the target audio is concatenated to the audio to be converted to obtain the target video; wherein, feature clustering refers to feature processing on training samples;

[0129] The acoustic model is obtained through training by feature clustering.

[0130] As can be seen from the above technical solutions, this application provides a video conversion method and system based on feature clustering. The method extracts the audio to be converted from the video to be converted; based on the audio text of the audio text, the language information of the audio text is a first language; language conversion is performed on the audio text according to a second language to obtain the text to be converted; the text to be converted and a reference audio are input into an acoustic model, which performs speech prediction based on the text features and the reference audio, outputs the target audio, and replaces the audio to be converted with the target audio to obtain the target video. This application, through speech clustering extraction, extracts higher-precision audio features from the reference audio even with low sample sizes, thereby improving the audio accuracy of the target audio during synthesis and enhancing the speech conversion effect.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0132] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the foregoing exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be made based on the foregoing teachings. The selection and description of the above embodiments are for the purpose of better explaining the contents of this disclosure, thereby enabling those skilled in the art to better utilize the described embodiments.

Claims

1. A video conversion method based on feature clustering, characterized in that, include: Extract the audio from the video to be converted; Based on the audio to be converted, an audio text is obtained, wherein the language information of the audio text is the first language; The audio text is converted to a second language, which is different from the first language, to obtain the text to be converted. The text to be converted and the reference audio are input into the acoustic model, so that the acoustic model can perform speech prediction based on the text features and the reference audio features, and output the target audio. The reference audio is audio data from the same sound source as the audio to be converted; the text features are text features extracted from the text to be converted; the reference audio features are audio features obtained by performing speech extraction based on the reference audio; the acoustic model is obtained through training by feature clustering; wherein, feature clustering refers to feature processing on training samples; Replace the audio to be converted with the target audio to obtain the target video; The acoustic model includes an audio encoding module, a text embedding module, an audio prediction module, and an audio decoding module. The method for inputting the text to be converted and the reference audio into the acoustic model includes: The audio encoding module performs speech clustering extraction on the reference audio to obtain reference audio features, and the text embedding module extracts text features of the text to be converted. The reference audio features and the text features are input into the audio prediction module to obtain the target audio code; The target audio is obtained by decoding the target audio code using the audio decoding module. The audio prediction module includes a quantization unit, an autoregressive unit, and a non-autoregressive unit. The step of inputting the reference audio features and the text features into the audio prediction module includes: The reference audio feature is quantized a preset number of times by the quantization unit to obtain a first audio vector and a second audio vector. The first audio vector is the audio vector obtained by performing quantization on the reference audio feature for the first time, and the second audio vector is the audio vector obtained by performing quantization on the reference audio feature other than the first audio vector. The first audio vector and the text features are input into the autoregressive unit to perform speech prediction on the first audio vector based on the text features, and obtain a first prediction result; The second audio vector is input into the non-autoregressive unit to perform speech prediction on the second audio vector and obtain a second prediction result; By concatenating the first prediction result and the second prediction result, the target audio code is obtained.

2. The video conversion method based on feature clustering according to claim 1, characterized in that, The quantization unit includes a hierarchical structure formed by several quantization sub-units. The input of each quantization sub-unit is the output of the previous quantization sub-unit, and the output of each quantization sub-unit is the input of the next quantization sub-unit. Performing a quantization operation on the reference audio features through the quantization unit includes: The reference audio features are input into the hierarchical structure to perform the first quantization operation on the reference audio features through the first quantization subunit to obtain the first audio vector; The first audio vector is input to the second quantization subunit, and residual vector quantization is performed on the first audio vector through the quantization subunit to obtain the second audio vector.

3. The video conversion method based on feature clustering according to claim 1, characterized in that, The audio prediction module further includes a style extraction unit, which is configured to extract style features from the audio vector. The step of inputting the second audio vector into the non-autoregressive unit includes: The second audio vector is input into the style extraction unit to generate a style embedding matrix, which is used to characterize the vectors corresponding to different style features in the second audio vector. The attention weights of the style features are calculated using an attention mechanism; A style feature sequence is generated based on the attention weights, and the style feature sequence is input into the non-autoregressive unit.

4. The video conversion method based on feature clustering according to claim 1, characterized in that, The audio encoding module includes an audio encoder, which includes a feature encoding unit and a category mapping unit. The step of performing speech clustering extraction on the reference audio through the audio encoding module includes: The clustering features of the reference audio are extracted using the feature encoding unit. The reference audio features are obtained by performing category mapping on the features to be clustered according to the preset category code by the category mapping unit.

5. The video conversion method based on feature clustering according to claim 4, characterized in that, The method further includes: Obtain a first training dataset, which includes a preset number of first training data. The first training data is input into the audio encoder to be trained, so as to extract the first training feature of the first training data through the feature encoding unit. The first training feature is subjected to category mapping by the category mapping unit to obtain the second training feature; Calculate the feature loss between the second training feature and the audio feature label, where the audio feature label is used to characterize standard audio features; If the feature loss is less than or equal to the first loss threshold, the audio encoder is output according to the current parameters of the feature encoding unit and the current parameters of the category mapping unit. If the feature loss is greater than the first loss threshold, then the feature encoding unit and the category mapping unit are iteratively trained based on the first training data.

6. The video conversion method based on feature clustering according to claim 5, characterized in that, The audio encoder further includes a clustering module and a category encoding unit. The step of performing category mapping on the first training features through the category mapping unit includes: The category code is obtained through the category coding unit, and the category code is obtained by the clustering module after performing clustering based on the speech sample data. The category mapping unit performs category mapping on the first training feature according to the category encoding to obtain the second training feature.

7. The video conversion method based on feature clustering according to claim 6, characterized in that, The clustering module includes a feature extractor, and before the step of obtaining the category code through the category encoding unit, the method further includes: Obtain a second training dataset, which includes a preset number of second training data. The second training data is input into the clustering module to be trained, so that the third training feature of the second training data can be extracted by the feature extractor; Clustering classification is performed on the third training feature to generate the category code.

8. A video conversion system based on feature clustering, characterized in that, It includes an audio extraction module, a text acquisition module, a conversion module, and an acoustic model; The audio extraction module is configured to extract the audio to be converted from the video to be converted; The text acquisition module is configured to obtain audio text based on the audio to be converted, wherein the language information of the audio text is a first language; The conversion module is configured to convert the audio text to a second language different from the first language to obtain the text to be converted. The acoustic model is configured to perform speech prediction based on text features and reference audio features, and output target audio. The reference audio is audio data from the same sound source as the audio to be converted; the text features are text features extracted from the text to be converted; the reference audio features are audio features extracted by clustering based on the reference audio; the target audio is concatenated to the audio to be converted to obtain the target video; wherein, feature clustering refers to feature processing on training samples; the acoustic model is obtained through feature clustering training. The acoustic model includes an audio encoding module, a text embedding module, an audio prediction module, and an audio decoding module. The acoustic model is configured to perform speech prediction based on text features and reference audio features, and output target audio, including: The audio encoding module performs speech clustering extraction on the reference audio to obtain reference audio features, and the text embedding module extracts text features of the text to be converted. The reference audio features and the text features are input into the audio prediction module to obtain the target audio code; The target audio is obtained by decoding the target audio code using the audio decoding module. The audio prediction module includes a quantization unit, an autoregressive unit, and a non-autoregressive unit. The step of inputting the reference audio features and the text features into the audio prediction module to obtain the target audio encoding includes: The reference audio feature is quantized a preset number of times by the quantization unit to obtain a first audio vector and a second audio vector. The first audio vector is the audio vector obtained by performing quantization on the reference audio feature for the first time, and the second audio vector is the audio vector obtained by performing quantization on the reference audio feature other than the first audio vector. The first audio vector and the text features are input into the autoregressive unit to perform speech prediction on the first audio vector based on the text features, and obtain a first prediction result; The second audio vector is input into the non-autoregressive unit to perform speech prediction on the second audio vector and obtain a second prediction result; By concatenating the first prediction result and the second prediction result, the target audio code is obtained.

Images