Moving video generation method and device, equipment and storage medium

Abstract

The invention discloses an oral playing video generation method, device and equipment and a storage medium, and relates to the technical field of computers, and the disclosed method comprises the steps: receiving an oral playing audio sample and current text content; learning the oral audio sample through the initial character model to generate a target character model corresponding to the tone of the oral audio sample; and generating an oral video based on the current text content through the target person model. The target character model corresponding to the tone of the oral audio sample is generated in a mode of learning the oral audio sample by using the initial character model, so that compared with the prior art that only an oral video with a default style tone can be generated based on a text, the method and the device have the advantages that the time is saved; according to the invention, the tone of the initial character model can be adjusted through the oral broadcast audio sample, so that the user demand can be met, and the user experience is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for generating spoken video. Background Art

[0002] With the rapid development of science and technology, we can express content by building character models, such as digital humans, and generate spoken videos by inputting arbitrary text content into the character models, thus replacing real people and meeting the respective needs of users.

[0003] However, the timbre of the character model currently used is fixed, and only spoken-word videos with a default style timbre can be generated based on text. Once the timbre of the spoken-word video output by the character model conflicts with user needs, it will affect the user experience.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present application is to provide a method, device, equipment and storage medium for generating spoken-word videos, aiming to solve the technical problem that the prior art can only generate spoken-word videos with default style and tone based on text, which affects the user experience.

[0006] To achieve the above object, the present application provides a method for generating a spoken video, the method comprising:

[0007] Receive spoken audio samples and current text content;

[0008] The spoken audio sample is learned through an initial character model to generate a target character model corresponding to the timbre of the spoken audio sample;

[0009] Generate a spoken video based on the current text content through the target character model.

[0010] In one embodiment, the step of learning the spoken audio sample through the initial character model to generate a target character model corresponding to the timbre of the spoken audio sample includes:

[0011] Identifying sample timbre characteristics of the spoken audio sample;

[0012] Determining a feature similarity between the sample timbre feature and the model timbre feature of the initial character model;

[0013] The initial character model is adjusted according to the feature similarity to generate a target character model corresponding to the timbre of the spoken audio sample.

[0014] In one embodiment, the step of adjusting the initial character model according to the feature similarity to generate a target character model corresponding to the timbre of the spoken audio sample includes:

[0015] Determining whether the feature similarity reaches a preset similarity;

[0016] If yes, adding the sample timbre feature to the training data of the initial character model to obtain updated training data;

[0017] The initial character model is trained using the updated training data to generate a target character model corresponding to the timbre of the spoken audio sample.

[0018] In one embodiment, after the step of determining whether the feature similarity reaches a preset similarity, the method further includes:

[0019] If not, obtaining model timbre characteristics of multiple preset character models;

[0020] Determining a model adjustment parameter according to the sample timbre characteristics and the model timbre characteristics of each of the preset character models;

[0021] The initial character model is adjusted according to the model adjustment parameters to generate a target character model corresponding to the timbre of the spoken audio sample.

[0022] In one embodiment, the step of determining the model adjustment parameter according to the sample timbre characteristics and the model timbre characteristics of each of the preset character models includes:

[0023] Determining the feature similarity between the sample timbre feature and the model timbre feature of each of the preset character models;

[0024] Determining a minimum feature similarity among the feature similarities corresponding to the preset character models;

[0025] Obtaining a preset character model corresponding to the minimum feature similarity;

[0026] The model parameters of the preset character model corresponding to the minimum feature similarity are extracted as model adjustment parameters.

[0027] In one embodiment, the step of adjusting the initial character model according to the model adjustment parameter to generate a target character model corresponding to the timbre of the spoken audio sample includes:

[0028] Migrating the model adjustment parameters to the initial character model to obtain the character model to be optimized;

[0029] Acquire an audio data set corresponding to the spoken audio sample, wherein the feature similarity between the timbre feature of each audio data in the audio data set and the sample timbre feature of the spoken audio sample is lower than the preset similarity;

[0030] The character model to be optimized is optimized through the audio data set to generate a target character model corresponding to the timbre of the spoken audio sample.

[0031] In one embodiment, before the step of learning the spoken audio sample through the initial character model to generate a target character model corresponding to the timbre of the spoken audio sample, the method further includes:

[0032] Show multiple preset character models to the user:

[0033] Determining whether to receive a selection instruction input by the user;

[0034] If so, a corresponding preset character model is selected from each of the preset character models as the initial character model according to the selection instruction.

[0035] In one embodiment, after the step of determining whether to receive a selection instruction input by the user, the method further includes:

[0036] If not, extracting the character features of the current text content;

[0037] Acquire annotation information of each of the preset character models, wherein the annotation information is information characterizing features of the preset character models;

[0038] Matching the character features with each of the annotation information;

[0039] The successfully matched preset character model is obtained as the initial character model.

[0040] In one embodiment, the step of generating a spoken video based on the current text content by using the target character model includes:

[0041] Determining whether the current text content involves other character models;

[0042] If so, extracting the first speech information corresponding to the target character model and the second speech information corresponding to the other character models from the current text content;

[0043] Generate a first spoken video based on the first speech information by using the target person model;

[0044] A second spoken-word video is generated based on the second speech information through the other character model, and the first spoken-word video and the second spoken-word video constitute a spoken-word video in the form of a multi-person dialogue.

[0045] In one embodiment, before the step of receiving the spoken audio sample and the current text content, the method further includes:

[0046] Obtain image data and audio data of different characters;

[0047] Extracting visual features from the image data, and extracting timbre features from the audio data;

[0048] Associating the visual features and timbre features of the same person to obtain training data;

[0049] The initial neural network model is trained using the training data to obtain a preset character model corresponding to each of the characters.

[0050] In addition, to achieve the above-mentioned purpose, the present application also proposes a spoken video generation device, the device comprising:

[0051] A data receiving module is used to receive the spoken audio sample and the current text content;

[0052] A model adjustment module, used to learn the spoken audio sample through an initial character model to generate a target character model corresponding to the timbre of the spoken audio sample;

[0053] A video generation module is used to generate a spoken video based on the current text content through the target character model.

[0054] In one embodiment, the model adjustment module is further used to identify sample timbre features of the spoken audio sample;

[0055] The model adjustment module is further used to determine the feature similarity between the sample timbre feature and the model timbre feature of the initial character model;

[0056] The model adjustment module is further used to adjust the initial character model according to the feature similarity to generate a target character model corresponding to the timbre of the spoken audio sample.

[0057] In one embodiment, the model adjustment module is further used to determine whether the feature similarity reaches a preset similarity;

[0058] The model adjustment module is further configured to add the sample timbre feature to the training data of the initial character model to obtain updated training data;

[0059] The model adjustment module is also used to train the initial character model using the updated training data to generate a target character model corresponding to the timbre of the spoken audio sample.

[0060] In one embodiment, the model adjustment module is further used to obtain model timbre characteristics of multiple preset character models if no;

[0061] The model adjustment module is further used to determine the model adjustment parameters according to the sample timbre characteristics and the model timbre characteristics of each of the preset character models;

[0062] The model adjustment module is further used to adjust the initial character model according to the model adjustment parameters to generate a target character model corresponding to the timbre of the spoken audio sample.

[0063] In one embodiment, the model adjustment module is further used to determine the feature similarity between the sample timbre feature and the model timbre feature of each of the preset character models;

[0064] The model adjustment module is further used to determine the minimum feature similarity among the feature similarities corresponding to the preset character models;

[0065] The model adjustment module is further used to obtain a preset character model corresponding to the minimum feature similarity;

[0066] The model adjustment module is further used to extract model parameters of the preset character model corresponding to the minimum feature similarity as model adjustment parameters.

[0067] In one embodiment, the model adjustment module is further used to transfer the model adjustment parameters to the initial character model to obtain the character model to be optimized;

[0068] The model adjustment module is further used to obtain an audio data set corresponding to the spoken audio sample, wherein the feature similarity between the timbre feature of each audio data in the audio data set and the sample timbre feature of the spoken audio sample is lower than the preset similarity;

[0069] The model adjustment module is also used to optimize the character model to be optimized through the audio data set to generate a target character model corresponding to the timbre of the spoken audio sample.

[0070] In one embodiment, the model adjustment module is further used to display a plurality of preset character models to the user:

[0071] The model adjustment module is further used to determine whether to receive the selection instruction input by the user;

[0072] The model adjustment module is further configured to, if yes, select a corresponding preset character model from each of the preset character models as the initial character model according to the selection instruction.

[0073] In one embodiment, the model adjustment module is further used to extract the character features of the current text content if no;

[0074] The model adjustment module is further used to obtain the annotation information of each of the preset character models, wherein the annotation information is information characterizing the characteristics of the preset character model;

[0075] The model adjustment module is further used to match the character features with each of the annotation information;

[0076] The model adjustment module is also used to obtain a successfully matched preset character model as an initial character model.

[0077] In addition, to achieve the above-mentioned purpose, the present application also proposes a voice-over video generation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the voice-over video generation method as described above.

[0078] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the voice-over video generation method described above are implemented.

[0079] One or more technical solutions proposed in this application have at least the following technical effects:

[0080] The present application receives a spoken audio sample and the current text content; learns the spoken audio sample through an initial character model to generate a target character model corresponding to the timbre of the spoken audio sample; and generates a spoken video based on the current text content through the target character model. Since the present application generates a target character model corresponding to the timbre of the spoken audio sample by learning the spoken audio sample using an initial character model, compared to the prior art that can only generate a spoken video with a default style timbre based on text, the present application can adjust the timbre of the initial character model through the spoken audio sample, thereby meeting user needs and effectively improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0082] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0083] Figure 1 This is a flow chart of the first embodiment of the method for generating oral video of the present application;

[0084] Figure 2 This is a flow chart of the second embodiment of the method for generating oral video of the present application;

[0085] Figure 3 This is a flow chart of the third embodiment of the method for generating oral video of the present application;

[0086] Figure 4 This is a schematic diagram of the module structure of the oral video generation device of the present application;

[0087] Figure 5 A schematic diagram of the structure of a spoken video generation device suitable for implementing an embodiment of the present application.

[0088] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0089] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0090] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0091] The main solution of the embodiment of the present application is: the spoken video generation device receives the spoken audio sample and the current text content; the spoken audio sample is learned through the initial character model to generate a target character model corresponding to the timbre of the spoken audio sample; and the spoken video is generated based on the current text content through the target character model.

[0092] In the embodiments of the present application, for ease of description, the following description is made using a spoken video generation device as the execution entity.

[0093] Since the timbre of the character model in the prior art is fixed, only spoken-word videos with a default style timbre can be generated based on the text. Once the timbre of the spoken-word video output by the character model conflicts with user needs, it will lead to a poor user experience.

[0094] The present application provides a solution, which generates a target character model corresponding to the timbre of the spoken audio sample by using the initial character model to learn the spoken audio sample. Compared with the existing technology that can only generate spoken videos with default style timbre based on text, the present application can adjust the timbre of the initial character model through the spoken audio sample, thereby meeting user needs and effectively improving the user experience.

[0095] It should be noted that the execution subject of this embodiment can be a computing service device with the functions of oral video generation, network communication and program running, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, an oral video generation device, etc. The following takes the oral video generation device as an example to illustrate this embodiment and the following embodiments.

[0096] Based on this, the embodiment of the present application provides a method for generating a spoken video, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the method for generating spoken video in the present application.

[0097] In this embodiment, the oral video generation method includes steps S10 to S30:

[0098] Step S10, receiving the spoken audio sample and the current text content.

[0099] It should be noted that the above-mentioned spoken audio sample may be a piece of audio data whose timbre is specified by the user.

[0100] It is understandable that the above current text content may be text data to be read aloud by the required character model.

[0101] The character model can be a digital, operable virtual character or digital human, which is used to generate a spoken video. The character model at least includes the visual image, voice characteristics and behavior patterns of the character.

[0102] In a specific implementation, the user can select or customize the audio content that meets his or her timbre preference as the spoken audio sample according to his or her preferences or specific needs, and input the spoken audio sample and the current text content obtained into the spoken video generation device. Or the user inputs the identification data representing the specified timbre, and the above-mentioned spoken video generation device can randomly generate a piece of audio content corresponding to the user's specified timbre as the spoken audio sample according to the identification data.

[0103] Step S20, learning the spoken audio sample through the initial character model to generate a target character model corresponding to the timbre of the spoken audio sample.

[0104] In a specific implementation, multiple character models can be constructed in advance through training as preset character models and saved in the spoken video generation device. The above-mentioned spoken video generation device can display the image of each preset character model on the corresponding interface, and the user can trigger a selection instruction based on the displayed image. The spoken video generation device can select the preset character model corresponding to the selection instruction from the preset character models as the initial character model according to the selection instruction.

[0105] Furthermore, the above-mentioned spoken-word video generation device can add the spoken-word audio sample as new audio data to the training data of the initial character model, and then use the training data of the spoken-word audio sample to train the initial character model, so that the initial character model learns the timbre in the spoken-word audio sample, and generates a target character model corresponding to the timbre of the spoken-word audio sample.

[0106] In a feasible implementation manner, before step S10, steps S01 to S04 may also be included:

[0107] Step S01, obtaining image data and audio data of different characters.

[0108] In a specific implementation, the above-mentioned voice-over video generation device can respond to the model building instructions triggered by the developer, obtain image data (such as photos, video frames) and audio data (such as voice recordings) of different characters from the picture library or online resources through the corresponding interface, or receive image data and audio data obtained by the developer using a camera and microphone for real-time recording.

[0109] Step S02: extracting visual features from the image data and extracting timbre features from the audio data.

[0110] It should be noted that the above-mentioned visual features may be features for describing or identifying a person's appearance, including at least facial features (such as the shape and relative position of the eyes, nose, mouth, face shape, etc.), body features (such as height, weight, body shape, posture, etc.) and appearance features (such as hairstyle, hair color, eye color, etc.), skin features (such as skin color, skin texture, wrinkles, scars, etc.).

[0111] It is understandable that the above-mentioned timbre features may be features for describing or identifying a person's voice, including at least the pitch, spectrum distribution, and intensity of the voice.

[0112] In a specific implementation, the convolutional neural network model can be trained using labeled training data. During the training process, the convolutional neural network model can learn the function of extracting visual features from image data. After receiving the real-time input image data and audio data, the above-mentioned oral video generation device can use the trained convolutional neural network to extract the visual features of each character from the image data, and at the same time, it can extract the timbre features of each character from the audio data.

[0113] Step S03, associating the visual features and timbre features of the same person to obtain training data.

[0114] In a specific implementation, the above-mentioned spoken video generation device can associate the visual features and timbre features belonging to the same character, and use the mutually associated visual features and timbre features as the training data of the character. By repeating the above process, the training data of each character can be obtained.

[0115] Step S04, training the initial neural network model using the training data to obtain a preset character model corresponding to each of the characters.

[0116] In a specific implementation, the above-mentioned spoken video generation device can obtain a multimodal neural network model that can simultaneously process image data and audio data, use the multimodal neural network model as the initial neural network model, and then use the training data of each character to train the initial neural network model to obtain a preset character model corresponding to each character.

[0117] Step S30, generating a spoken video based on the current text content through the target character model.

[0118] In a specific implementation, the above-mentioned spoken video generation device can input the current text content into the target character model, and the target character model converts the input current text content into a speech waveform according to the learned timbre (i.e., the timbre corresponding to the spoken audio sample), and combines it with the virtual image corresponding to the target character model to generate a spoken video in which the virtual image reads the current text content in the timbre corresponding to the spoken audio sample.

[0119] This embodiment receives a spoken audio sample and the current text content; learns the spoken audio sample through an initial character model to generate a target character model corresponding to the timbre of the spoken audio sample; and generates a spoken video based on the current text content through the target character model. Since this embodiment generates a target character model corresponding to the timbre of the spoken audio sample by learning the spoken audio sample using an initial character model, compared to the prior art that can only generate a spoken video with a default style timbre based on text, this embodiment can adjust the timbre of the initial character model through the spoken audio sample, thereby meeting user needs and effectively improving the user experience.

[0120] Based on the first embodiment of the present application, the second embodiment of the present application is proposed. In the second embodiment of the present application, the same or similar contents as those of the first embodiment can be referred to the above introduction, and will not be repeated later. Figure 2 , Figure 2 This is a flow chart of the second embodiment of the method for generating spoken video of the present application.

[0121] In this embodiment, step S20 may include steps S201 to S203:

[0122] Step S201, identifying the sample timbre characteristics of the spoken audio sample.

[0123] In a specific implementation, the above-mentioned spoken video generation device can divide the original audio signal of the spoken audio sample into several short frames, each frame containing several milliseconds (such as 20 to 30 milliseconds) of audio data, apply a window function (such as a Hamming window) to each frame of audio data for windowing, and perform fast Fourier transform on the windowed audio data, so as to convert each frame of windowed audio data into a spectrum signal, and then filter each spectrum signal through a group of filter groups, perform logarithmic operation and discrete cosine transform on the filtered spectrum data in turn, and then identify the audio features in the transformed data as the sample audio features of the spoken audio sample.

[0124] Step S202: determining the feature similarity between the sample timbre feature and the model timbre feature of the initial character model.

[0125] In a specific implementation, the above-mentioned spoken video generation device can obtain the timbre features learned by the initial character model from the training data as the model timbre features of the initial character model, and then convert the sample timbre features into corresponding sample timbre feature vectors, and convert the model timbre features into corresponding model timbre feature vectors, and calculate the feature similarity between the sample timbre features and the model timbre features through the sample timbre feature vectors and the model timbre feature vectors.

[0126] Step S203, adjusting the initial character model according to the feature similarity to generate a target character model corresponding to the timbre of the spoken audio sample.

[0127] In a specific implementation, the above-mentioned spoken video generation device can compare the feature similarity with a pre-set similarity calibration value after obtaining the feature similarity. If the feature similarity is lower than the similarity calibration value, the model parameters of the initial character model, such as weights, bias items, hyperparameters, etc., are adjusted, and then any text is randomly generated, and a spoken video of the any text is output through the initial character model. When the feature similarity between the timbre characteristics of the spoken video and the timbre characteristics of the sample is lower than the similarity calibration value, the above process is repeated until the feature similarity between the timbre characteristics of the spoken video output by the initial character model and the sample timbre characteristics is lower than the similarity calibration value, and the adjusted initial character model is used as the target character model corresponding to the timbre of the spoken audio sample.

[0128] In a feasible implementation, step S203 may include steps S21 to S23:

[0129] Step S21, determining whether the feature similarity reaches a preset similarity.

[0130] It should be noted that the above-mentioned preset similarity can be the above-mentioned similarity calibration value, which is used to indicate whether the sample timbre feature is close to the model timbre feature.

[0131] In a specific implementation, the above-mentioned oral video generation device can compare the feature similarity with the preset similarity to determine whether the feature similarity reaches the preset similarity. When the feature similarity reaches the preset similarity, it is determined that the sample timbre feature is close to the model timbre feature, that is, the timbre of the initial character model is close to the timbre of the oral audio sample. On the contrary, when the feature similarity does not reach the preset similarity, it is determined that there is a large difference between the sample timbre feature and the model timbre feature, that is, there is a large difference between the timbre of the initial character model and the timbre of the oral audio sample.

[0132] Step S22: If yes, then the sample timbre feature is added to the training data of the initial character model to obtain updated training data.

[0133] In a specific implementation, when the above-mentioned spoken video generation device detects that the feature similarity reaches a preset similarity, it determines that the timbre of the initial character model is relatively close to the timbre of the spoken audio sample. In order to further improve the adjustment accuracy of the initial character model, the sample timbre features can be added as new audio data to the training data of the initial character model to update the training data and obtain updated training data.

[0134] Step S23, training the initial character model using the updated training data to generate a target character model corresponding to the timbre of the spoken audio sample.

[0135] In a specific implementation, the above-mentioned spoken video generation device can retrain the initial character model through the updated training data. Since the updated training data contains the sample timbre characteristics, the obtained target character model can learn the timbre of the spoken audio sample, making the timbre of the target character model closer to the timbre of the spoken audio sample, effectively improving the adjustment accuracy of the initial character model.

[0136] In a feasible implementation manner, step S21 may further include steps S24 to S26:

[0137] Step S24: if not, then obtaining model timbre features of multiple preset character models.

[0138] In a specific implementation, when the above-mentioned spoken video generation device detects that the feature similarity does not reach the preset similarity, it determines that there is a large difference between the timbre of the initial character model and the timbre of the spoken audio sample. At this time, the model timbre features of multiple preset character models can be obtained, and the preset character models obtained are character models other than the initial character model.

[0139] Step S25 , determining a model adjustment parameter according to the sample timbre characteristics and the model timbre characteristics of each of the preset character models.

[0140] In a specific implementation, the above-mentioned spoken video generation device can compare the model timbre characteristics of each preset character model with the sample timbre characteristics, select the preset character model corresponding to the model timbre characteristics with smaller feature differences, and extract the model parameters of the selected preset character model as model adjustment parameters.

[0141] In a feasible implementation, step S25 may include steps S251 to S254:

[0142] Step S251, determining the feature similarity between the sample timbre feature and the model timbre feature of each of the preset character models.

[0143] In a specific implementation, the above-mentioned spoken video generation device can calculate the feature similarity between the sample timbre features and the model timbre features of each preset character model, and use each feature similarity as the feature difference between the sample timbre features and the model timbre features of each preset character model.

[0144] Step S252: determining a minimum feature similarity among the feature similarities corresponding to the preset character models.

[0145] In a specific implementation, the above-mentioned spoken video generation device can traverse each feature similarity after obtaining the feature similarity between the sample timbre feature and the model timbre feature of each preset character model, and determine the similarity with the smallest value among the feature similarities, that is, the minimum feature similarity.

[0146] Step S253, obtaining a preset character model corresponding to the minimum feature similarity.

[0147] In a specific implementation, the above-mentioned spoken video generation device can obtain the preset character model with the smallest similarity with the sample features from various preset character models, that is, the preset character model corresponding to the smallest feature similarity.

[0148] Step S254: extracting the model parameters of the preset character model corresponding to the minimum feature similarity as model adjustment parameters.

[0149] In a specific implementation, the above-mentioned spoken video generation device can extract model parameters of the preset character model corresponding to the minimum feature similarity, such as weights, bias items, hyperparameters, etc., after determining the preset character model corresponding to the minimum feature similarity.

[0150] Step S26, adjusting the initial character model according to the model adjustment parameters to generate a target character model corresponding to the timbre of the spoken audio sample.

[0151] In a specific implementation, the above-mentioned spoken video generation device can apply the above-mentioned model feature parameters to the initial character model, such as replacing the original parameters at the same position in the initial character model, to obtain a target character model corresponding to the timbre of the spoken audio sample.

[0152] In a feasible implementation, step S26 may include steps S261 to S263:

[0153] Step S261, transferring the model adjustment parameters to the initial character model to obtain the character model to be optimized.

[0154] In a specific implementation, the above-mentioned spoken video generation device can migrate the model feature parameters to the initial character model, such as replacing the original parameters at the same position in the initial character model with the model feature parameters. In order to improve the generation accuracy of the target character model, the character model obtained at this time can be used as the character model to be optimized for subsequent optimization operations.

[0155] Step S262, obtaining an audio data set corresponding to the spoken audio sample, wherein the feature similarity between the timbre features of each audio data in the audio data set and the sample timbre features of the spoken audio sample is lower than the preset similarity.

[0156] In a specific implementation, the above-mentioned spoken video generation device can obtain multiple audio data whose feature similarity with the sample timbre features of the spoken audio sample is lower than the preset similarity from a picture library or online resources, and add the obtained audio data to the same data set to obtain an audio data set corresponding to the spoken audio sample.

[0157] Step S263, optimizing the character model to be optimized by using the audio data set, and generating a target character model corresponding to the timbre of the spoken audio sample.

[0158] In a specific implementation, the above-mentioned spoken video generation device can train the character model to be optimized through an audio data set to optimize the character model to be optimized. Since the feature similarity between the timbre features of each audio data in the audio data set and the sample timbre features of the spoken audio sample is lower than the preset similarity, the timbre features of the generated target character model can be made closer to the timbre of the spoken audio sample, thereby improving the generation accuracy of the target character model.

[0159] This embodiment identifies the sample timbre features of the spoken audio sample, determines the feature similarity between the sample timbre features and the model timbre features of the initial character model, and when the feature similarity reaches a preset similarity, adds the sample timbre features to the training data of the initial character model and then trains the initial character model to generate a target character model corresponding to the timbre of the spoken audio sample; when the feature similarity does not reach the preset similarity, determines the model adjustment parameters according to the sample timbre features and the model timbre features of each preset character model, adjusts the initial character model according to the model adjustment parameters, and generates a target character model corresponding to the timbre of the spoken audio sample, thereby achieving different strategies for generating the target character model according to the size of the feature similarity and the preset similarity, and effectively improving the generation accuracy of the target character model.

[0160] Based on the first and second embodiments of the present application, the third embodiment of the present application is proposed. In the third embodiment of the present application, the same or similar contents as those of the first and second embodiments can be referred to the above introduction, and will not be repeated in the following. Figure 3 , Figure 3 This is a flow chart of the third embodiment of the method for generating spoken video of the present application.

[0161] In this embodiment, before step S21, steps S11 to S13 may also be included:

[0162] Step S11, displaying multiple preset character models to the user.

[0163] In a specific implementation, the above-mentioned spoken video generation device can display various preset character models on the corresponding interface for users to choose.

[0164] Step S12, determining whether a selection instruction input by the user is received.

[0165] In a specific implementation, the above-mentioned spoken video generation device can determine whether the user inputs a selection instruction to confirm whether the user expects to generate a spoken video using the preset character model specified by the user.

[0166] Step S13: If yes, then select a corresponding preset character model from each of the preset character models as the initial character model according to the selection instruction.

[0167] In a specific implementation, the above-mentioned spoken-word video generation device can select a preset character model corresponding to the selection instruction from the displayed preset character models as the initial character model when receiving the user's input spoken-word audio sample, the current text content, and a selection instruction based on feedback from multiple preset character models, and then use the initial character model to learn the spoken-word audio sample to obtain the target character model, and use the target character model to generate a spoken-word video corresponding to the current text content.

[0168] In a feasible implementation manner, step S12 may further include steps S121 to S124:

[0169] Step S121: if not, extracting the character features of the current text content.

[0170] It should be noted that the above-mentioned character characteristics may be descriptive information related to the character, such as the character's gender, age, occupation, personality, appearance description, etc.

[0171] In a specific implementation, the above-mentioned spoken video generation device can traverse the current text content and extract character features related to the character description in the current text content when it receives the user input spoken audio sample and the current text content but does not select instructions based on multiple preset character model feedback.

[0172] Step S122, obtaining the annotation information of each of the preset character models, wherein the annotation information is information representing the characteristics of the preset character models.

[0173] It should be noted that the above-mentioned annotation information may be information pre-configured for a preset character model and identifying character features of the preset character model.

[0174] Step S123, matching the character features with each of the annotation information.

[0175] In a specific implementation, after each preset character model is trained, corresponding annotation information can be configured for each preset character model in advance. The above-mentioned spoken video generation device can determine the similarity between the character features corresponding to the current text content and the annotation information of each preset character model for matching.

[0176] Step S124, obtaining a successfully matched preset character model as an initial character model.

[0177] In a specific implementation, the above-mentioned spoken video generation device can select a preset character model that reaches a set similarity threshold as a successfully matched initial character model. When the similarity between the character features and the annotation information of each preset character model is lower than the similarity threshold, the preset character model with the highest similarity is selected as the successfully matched initial character model. The initial character model is then used to learn the timbre of the spoken audio sample to obtain a target character model, and the target character model is used to generate a spoken video corresponding to the current text content.

[0178] In this embodiment, step S30 may further include steps S301 to S304:

[0179] Step S301, determining whether the current text content involves other character models.

[0180] In a specific implementation, the current text content input by the user may be a text involving a conversation between multiple people. After determining the initial character model, the above-mentioned spoken video generation device may further determine whether the current text content involves multiple character conversations to determine whether the current text content involves other character models in order to generate a spoken video in the form of a conversation.

[0181] Among them, when the current text content does not involve multiple character dialogues, the initial character model can be directly used to learn the timbre of the spoken audio sample to obtain the target character model, and the target character model can be used to generate a spoken video corresponding to the current text content, that is, the current text content is read aloud by the virtual image corresponding to the target character model.

[0182] Step S302: If yes, extracting the first speech information corresponding to the target character model and the second speech information corresponding to the other character models from the current text content.

[0183] In a specific implementation, the user can feed back the current text content in which multiple characters participate to the oral video generation device, and at the same time, the corresponding oral audio sample can be input for each character. The user can specify several preset character models among multiple preset character models, and specify the oral audio sample corresponding to each preset character model, triggering the corresponding selection instruction. When the above-mentioned oral video generation device detects that the current text content involves multiple character dialogues, it determines that the current text content involves other character models. At this time, the above-mentioned oral video generation device can obtain all preset character models corresponding to the selection instruction from each preset character model according to the selection instruction, use any selected preset character model as the initial character model, and use the remaining preset character models as other preset character models. At the same time, according to the selection instruction, determine the oral audio sample corresponding to the initial character model, and the oral audio sample corresponding to the other preset character models, and then use each initial character model to learn the timbre of the corresponding oral audio sample to obtain the target character model corresponding to each initial character model, and use other preset character models to learn the timbre of the corresponding oral audio sample to obtain other character models corresponding to other preset character models.

[0184] Furthermore, when the user inputs the current text content and a single spoken audio sample, and has not selected any preset character models, the above-mentioned spoken video generation device can extract all character features in the current text content, match each character feature with the annotation information of each preset character model, and randomly select any preset character model from the preset character models that are successfully matched as the initial character model, and use the remaining preset character models as other preset character models. At the same time, the initial character model is used to learn the timbre of the spoken audio sample to obtain the target character model. Other preset character models do not use spoken audio samples for learning, that is, other preset character models can be other character models.

[0185] Correspondingly, after obtaining the target character model corresponding to the initial character model, the above-mentioned spoken video generation device can determine the dialogue text generated by the character corresponding to the target character model in the current text content as the first speech information, and the dialogue information generated by the characters corresponding to other character models as the second speech information.

[0186] Step S303: Generate a first spoken video based on the first speech information using the target person model.

[0187] In a specific implementation, the above-mentioned spoken video generation device can input the first speech information into the target character model, and the target character model converts the input first speech information into a speech waveform, and combines it with the virtual image corresponding to the target character model to generate the first spoken video.

[0188] Step S304: Generate a second spoken-word video based on the second speech information through the other character model, and the first spoken-word video and the second spoken-word video constitute a spoken-word video in the form of a multi-person dialogue.

[0189] In a specific implementation, the above-mentioned spoken video generation device can input the second speech information into other character models, and the other character models will convert the input second speech information into a voice waveform, and combine it with the virtual image corresponding to the other character model to generate a second spoken video. The spoken video finally generated by the spoken video generation device can form a video in the form of a multi-person dialogue composed of the first spoken video and the second spoken video.

[0190] When receiving a selection instruction input by the user, this embodiment obtains the preset character model corresponding to the selection instruction as the initial character model. When not receiving a selection instruction input by the user, the initial character model is determined by matching the character features of the current text content with the annotation information of each preset character model, thereby effectively improving the accuracy of determining the initial character model. Furthermore, when the current text content involves other character models, this embodiment generates a first spoken video based on the first speech information through the target character model, and generates a second spoken video based on the second speech information through other character models, thereby generating a spoken video in the form of a multi-person dialogue consisting of the first spoken video and the second spoken video, effectively improving the user experience.

[0191] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the oral video generation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0192] This application also provides a spoken video generation device, please refer to Figure 4 , Figure 4 This is a schematic diagram of the module structure of the oral video generation device of the present application, and the oral video generation device includes:

[0193] The data receiving module 10 is used to receive the spoken audio sample and the current text content;

[0194] A model adjustment module 20, configured to learn the spoken audio sample through an initial character model to generate a target character model corresponding to the timbre of the spoken audio sample;

[0195] The video generation module 30 is used to generate a spoken video based on the current text content through the target character model.

[0196] The oral video generation device provided by the present application adopts the oral video generation method in the above embodiment, which can solve the technical problem that the prior art can only generate oral videos of default style and tone based on text, affecting the user experience. Compared with the prior art, the beneficial effects of the oral video generation device provided by the present application are the same as the beneficial effects of the oral video generation method provided by the above embodiment, and the other technical features in the oral video generation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0197] As an implementation mode, the model adjustment module 20 is further used to identify the sample timbre characteristics of the spoken audio sample;

[0198] The model adjustment module 20 is further used to determine the feature similarity between the sample timbre feature and the model timbre feature of the initial character model;

[0199] The model adjustment module 20 is further used to adjust the initial character model according to the feature similarity to generate a target character model corresponding to the timbre of the spoken audio sample.

[0200] As an implementation manner, the model adjustment module 20 is further used to determine whether the feature similarity reaches a preset similarity;

[0201] The model adjustment module 20 is further configured to add the sample timbre feature to the training data of the initial character model to obtain updated training data;

[0202] The model adjustment module 20 is further used to train the initial character model using the updated training data to generate a target character model corresponding to the timbre of the spoken audio sample.

[0203] As an implementation mode, the model adjustment module 20 is further used to obtain the model timbre characteristics of multiple preset character models if no;

[0204] The model adjustment module 20 is further used to determine the model adjustment parameters according to the sample timbre characteristics and the model timbre characteristics of each of the preset character models;

[0205] The model adjustment module 20 is further used to adjust the initial character model according to the model adjustment parameters to generate a target character model corresponding to the timbre of the spoken audio sample.

[0206] As an implementation manner, the model adjustment module 20 is further used to determine the feature similarity between the sample timbre feature and the model timbre feature of each of the preset character models;

[0207] The model adjustment module 20 is further used to determine the minimum feature similarity among the feature similarities corresponding to the preset character models;

[0208] The model adjustment module 20 is further used to obtain a preset character model corresponding to the minimum feature similarity;

[0209] The model adjustment module 20 is further used to extract model parameters of the preset character model corresponding to the minimum feature similarity as model adjustment parameters.

[0210] As an implementation manner, the model adjustment module 20 is further used to transfer the model adjustment parameters to the initial character model to obtain the character model to be optimized;

[0211] The model adjustment module 20 is further used to obtain an audio data set corresponding to the spoken audio sample, wherein the feature similarity between the timbre feature of each audio data in the audio data set and the sample timbre feature of the spoken audio sample is lower than the preset similarity;

[0212] The model adjustment module 20 is further used to optimize the character model to be optimized through the audio data set to generate a target character model corresponding to the timbre of the spoken audio sample.

[0213] As an implementation manner, the model adjustment module 20 is further used to display a plurality of preset character models to the user:

[0214] The model adjustment module 20 is further used to determine whether to receive the selection instruction input by the user;

[0215] The model adjustment module 20 is further configured to select a corresponding preset character model from each of the preset character models as the initial character model according to the selection instruction.

[0216] As an implementation mode, the model adjustment module 20 is further used to extract the character features of the current text content if no;

[0217] The model adjustment module 20 is further used to obtain the annotation information of each of the preset character models, wherein the annotation information is information characterizing the characteristics of the preset character model;

[0218] The model adjustment module 20 is further used to match the character features with each of the annotation information;

[0219] The model adjustment module 20 is further used to obtain a successfully matched preset character model as an initial character model.

[0220] As an implementation mode, the video generation module 30 is further used to determine whether the current text content involves other character models;

[0221] The video generation module 30 is further configured to extract, from the current text content, first speech information corresponding to the target character model and second speech information corresponding to the other character models;

[0222] The video generation module 30 is further configured to generate a first spoken video based on the first speech information by using the target person model;

[0223] The video generation module 30 is further used to generate a second spoken video based on the second speech information through the other character models, and the first spoken video and the second spoken video constitute a spoken video in the form of a multi-person dialogue.

[0224] As an implementation mode, the model feature module 20 is also used to obtain image data and audio data of different characters;

[0225] The model feature module 20 is also used to extract visual features from the image data and to extract timbre features from the audio data;

[0226] The model feature module 20 is also used to associate the visual features and timbre features of the same person to obtain training data;

[0227] The model feature module 20 is also used to train the initial neural network model through the training data to obtain a preset character model corresponding to each of the characters.

[0228] The present application provides a spoken video generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the spoken video generation method in the above-mentioned embodiment 1.

[0229] Reference below Figure 5 , Figure 5 The structure diagram of the oral video generation device suitable for implementing the embodiment of the present application is shown in FIG. The oral video generation device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5The voice-over video generation device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0230] like Figure 5 As shown, the oral video generation device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the oral video generation device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the spoken video generation device to communicate wirelessly or wired with other devices to exchange data. Although the spoken video generation device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0231] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0232] The oral video generation device provided by the present application adopts the oral video generation method in the above embodiment, which can solve the technical problem that the prior art can only generate oral videos of default style and tone based on text, affecting the user experience. Compared with the prior art, the beneficial effects of the oral video generation device provided by the present application are the same as the beneficial effects of the oral video generation method provided by the above embodiment, and the other technical features in the oral video generation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0233] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0234] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0235] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the spoken video generation method in the above-mentioned embodiment.

[0236] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0237] The above-mentioned computer-readable storage medium may be included in the oral video generation device; or it may exist independently without being assembled in the oral video generation device.

[0238] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the spoken-word video generating device, the spoken-word video generating device: receives a spoken-word audio sample and a current text content; learns the spoken-word audio sample through an initial character model to generate a target character model corresponding to the timbre of the spoken-word audio sample; and generates a spoken-word video based on the current text content through the target character model.

[0239] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0240] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0241] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0242] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned spoken video generation method, and can solve the technical problem that the prior art can only generate spoken videos of default style and tone based on text, which affects the user experience. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the spoken video generation method provided by the above-mentioned embodiment, and will not be repeated here.

[0243] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for generating a spoken video, characterized in that: The method comprises: Receive spoken audio samples and current text content; The spoken audio sample is learned through an initial character model to generate a target character model corresponding to the timbre of the spoken audio sample; Generate a spoken video based on the current text content through the target character model.

2. The method for generating oral video according to claim 1, characterized in that: The step of learning the spoken audio sample through the initial character model to generate a target character model corresponding to the timbre of the spoken audio sample includes: Identifying sample timbre characteristics of the spoken audio sample; Determining a feature similarity between the sample timbre feature and the model timbre feature of the initial character model; The initial character model is adjusted according to the feature similarity to generate a target character model corresponding to the timbre of the spoken audio sample.

3. The method for generating oral video according to claim 2, characterized in that: The step of adjusting the initial character model according to the feature similarity to generate a target character model corresponding to the timbre of the spoken audio sample includes: Determining whether the feature similarity reaches a preset similarity; If yes, adding the sample timbre feature to the training data of the initial character model to obtain updated training data; The initial character model is trained using the updated training data to generate a target character model corresponding to the timbre of the spoken audio sample.

4. The method for generating oral video according to claim 3, characterized in that: After the step of determining whether the feature similarity reaches a preset similarity, the method further includes: If not, obtaining model timbre characteristics of multiple preset character models; Determining a model adjustment parameter according to the sample timbre characteristics and the model timbre characteristics of each of the preset character models; The initial character model is adjusted according to the model adjustment parameters to generate a target character model corresponding to the timbre of the spoken audio sample.

5. The method for generating oral video according to claim 1, characterized in that: Before the step of learning the spoken audio sample through the initial character model to generate a target character model corresponding to the timbre of the spoken audio sample, the method further includes: Show multiple preset character models to the user: Determining whether to receive a selection instruction input by the user; If so, a corresponding preset character model is selected from each of the preset character models as the initial character model according to the selection instruction.

6. The method for generating oral video according to claim 5, characterized in that: After the step of determining whether to receive the selection instruction input by the user, the method further includes: If not, extracting the character features of the current text content; Acquire annotation information of each of the preset character models, wherein the annotation information is information characterizing features of the preset character models; Matching the character features with each of the annotation information; The successfully matched preset character model is obtained as the initial character model.

7. The method for generating oral video according to any one of claims 1 to 5, characterized in that: The step of generating a spoken video based on the current text content by using the target character model includes: Determining whether the current text content involves other character models; If so, extracting the first speech information corresponding to the target character model and the second speech information corresponding to the other character models from the current text content; Generate a first spoken video based on the first speech information by using the target person model; A second spoken-word video is generated based on the second speech information through the other character model, and the first spoken-word video and the second spoken-word video constitute a spoken-word video in the form of a multi-person dialogue.

8. A spoken video generation device, characterized in that: The device comprises: A data receiving module is used to receive the spoken audio sample and the current text content; A model adjustment module, used to learn the spoken audio sample through an initial character model to generate a target character model corresponding to the timbre of the spoken audio sample; A video generation module is used to generate a spoken video based on the current text content through the target character model.

9. A spoken video generation device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method for generating a spoken video as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the spoken video generation method as described in any one of claims 1 to 7 are implemented.

Images