Digital mouth lip training method and device based on sound cloning, equipment and medium

Abstract

The invention belongs to the technical field of artificial intelligence, and discloses a digital human lip training method and device based on sound cloning, equipment and a medium, and the method comprises the steps: obtaining a pre-trained multi-mode voice model and to-be-cloned text data, inputting the second voice data of the target object and the to-be-cloned text data into a pre-trained multi-mode voice model for training, and generating cloned voice data; constructing a digital human lip action prediction model based on the constructed mapping relation between the voice data of the target object and the mouth lip action; inputting the generated cloned voice data into a digital human lip action prediction model, and predicting various parameter information of the digital human lip action; and based on the predicted parameter information of the lip action of the digital person, driving the digital person to simulate the lip action presented when the target object outputs the cloned voice data. According to the invention, the accuracy of the digital lip training is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a digital mouth and lip training method, device, equipment and medium based on sound cloning. Background Art

[0002] At present, when digital humans are trained on their lips, most of the time, videos of real people are used as training materials, and audio and facial information are extracted from the videos of real people to train the digital humans on their lips. However, as people's requirements for customized digital humans are increasing, the demand for "thousands of faces for thousands of people" is growing, and relying solely on these existing materials can no longer meet actual needs. In addition, when synthesizing digital humans, although lip movements can be generated with the help of sound, they can only rely on a limited number of video materials in reality, and it is difficult to dig out the lip shapes of individuals in different emotional states, which makes the accuracy of digital human lip training low.

[0003] In view of this, how to improve the accuracy of digital population lip training is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides a digital mouth and lip training method, device, equipment and medium based on sound cloning, so as to solve the technical problem that the accuracy of the existing digital mouth and lip training is not high.

[0005] In a first aspect, the present invention provides a digital mouth and lip training method based on sound cloning, comprising:

[0006] Acquire first voice data and a corresponding first lip movement video of a target object, and perform data preprocessing on the acquired first voice data and the corresponding first lip movement video of the target object to obtain second voice data and a corresponding second lip movement video of the target object;

[0007] Based on the second voice data of the target object and the corresponding second lip movement video, construct a mapping relationship between the voice data and the lip movement of the target object;

[0008] Acquire a pre-trained multimodal voice model and text data to be cloned, and input the second voice data of the target object and the text data to be cloned into the pre-trained multimodal voice model for training to generate cloned voice data;

[0009] Based on the mapping relationship between the speech data and lip movements of the constructed target object, a digital human lip movement prediction model is constructed;

[0010] Input the generated cloned voice data into the digital mouth lip movement prediction model to predict various parameter information of the digital mouth lip movement;

[0011] Based on the predicted parameters of the digital human's lip movements, the digital human is driven to simulate the lip movements of the target object when outputting cloned voice data.

[0012] In a second aspect, the present invention provides a digital mouth and lip training device based on sound cloning, the device is used to implement the digital mouth and lip training method based on sound cloning as described in the first aspect, comprising:

[0013] a processing module, configured to obtain first voice data of a target object and a corresponding first lip movement video, and perform data preprocessing on the obtained first voice data of the target object and the corresponding first lip movement video to obtain second voice data of the target object and a corresponding second lip movement video;

[0014] A first construction module is used to construct a mapping relationship between the voice data and the lip movement of the target object based on the second voice data of the target object and the corresponding second lip movement video;

[0015] A training module, used to obtain a pre-trained multimodal voice model and text data to be cloned, and input the second voice data of the target object and the text data to be cloned into the pre-trained multimodal voice model for training to generate cloned voice data;

[0016] The second construction module is used to construct a digital human lip movement prediction model based on the mapping relationship between the constructed target object's speech data and lip movements;

[0017] A prediction module, used for inputting the generated cloned voice data into a digital human mouth lip movement prediction model to predict various parameter information of the digital human mouth lip movement;

[0018] The driving module is used to drive the lip movements presented by the digital human to simulate the target object when outputting cloned voice data based on the predicted parameter information of the digital human's lip movements.

[0019] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned digital lip training method based on sound cloning when executing the computer program.

[0020] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-mentioned digital lip training method based on sound cloning.

[0021] In the scheme implemented by the above-mentioned digital lip training method, device, equipment and medium based on sound cloning, the first voice data and the corresponding first lip movement video of the target object can be obtained through the client, and the obtained first voice data and the corresponding first lip movement video of the target object are preprocessed to obtain the second voice data and the corresponding second lip movement video of the target object; based on the second voice data and the corresponding second lip movement video of the target object, a mapping relationship between the voice data and the lip movement of the target object is constructed; a pre-trained multimodal voice model and the text data to be cloned are obtained, and the second voice data and the text data to be cloned of the target object are input into the pre-trained multimodal voice model for training to generate cloned voice data; based on the constructed voice data and lip movement video of the target object The mapping relationship between lip movements is used to construct a digital human mouth lip movement prediction model; the generated cloned voice data is input into the digital human mouth lip movement prediction model to predict various parameter information of the digital human mouth lip movement; based on the predicted various parameter information of the digital human mouth lip movement, the digital human is driven to simulate the lip movement presented by the target object when outputting the cloned voice data. In the present invention, the text data to be cloned can be first input into a pre-trained multimodal voice model for training to generate cloned voice data, and then the generated cloned voice data can be input into the digital human mouth lip movement prediction model to predict various parameter information of the digital human mouth lip movement; finally, according to the predicted various parameter information of the digital human mouth lip movement, the digital human is driven to simulate the lip movement presented by the target object when outputting the cloned voice data, thereby effectively improving the accuracy of the digital human mouth lip training. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0023] Figure 1 2 is a schematic diagram of an application environment of a digital mouth and lip training method based on sound cloning in one embodiment of the present invention;

[0024] Figure 2 is a flow chart of a digital mouth and lip training method based on sound cloning in one embodiment of the present invention;

[0025] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S20;

[0026] Figure 4 yes Figure 2A schematic flow chart of a specific implementation of step S30;

[0027] Figure 5 yes Figure 4 A schematic flow chart of a specific implementation of step S34;

[0028] Figure 6 is a structural schematic diagram of a digital mouth and lip training device based on sound cloning in one embodiment of the present invention;

[0029] Figure 7 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0030] Figure 8 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] The digital mouth and lip training method based on sound cloning provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, Figure 1This is a schematic diagram of an application environment of a digital population lip training method based on sound cloning in one embodiment of the present invention; wherein, the client communicates with the server through a network. The server can obtain the first voice data and the corresponding first lip movement video of the target object through the client, and perform data preprocessing on the obtained first voice data and the corresponding first lip movement video of the target object to obtain the second voice data and the corresponding second lip movement video of the target object; based on the second voice data and the corresponding second lip movement video of the target object, construct a mapping relationship between the voice data and the lip movement of the target object; obtain a pre-trained multimodal voice model and the text data to be cloned, and input the second voice data and the text data to be cloned of the target object into the pre-trained multimodal voice model for training to generate cloned voice data; based on the constructed mapping relationship between the voice data and the lip movement of the target object, construct a digital human voice model. Lip movement prediction model; input the generated cloned voice data into the digital human mouth lip movement prediction model to predict various parameter information of the digital human mouth lip movement; based on the predicted various parameter information of the digital human mouth lip movement, drive the digital human to simulate the lip movement presented by the target object when outputting the cloned voice data. In the present invention, the text data to be cloned can be first input into the pre-trained multimodal voice model for training to generate cloned voice data, and then the generated cloned voice data can be input into the digital human mouth lip movement prediction model to predict various parameter information of the digital human mouth lip movement. Finally, according to the predicted various parameter information of the digital human mouth lip movement, drive the digital human to simulate the lip movement presented by the target object when outputting the cloned voice data, thereby effectively improving the accuracy of the digital human mouth lip training. The present invention is described in detail below through specific embodiments.

[0033] See also Figure 2 As shown, Figure 2 A flow chart of a digital mouth and lip training method based on sound cloning provided in an embodiment of the present invention is provided. The digital mouth and lip training method based on sound cloning specifically comprises the following steps:

[0034] S10: Acquire the first voice data and the corresponding first lip movement video of the target object, and perform data preprocessing on the acquired first voice data and the corresponding first lip movement video of the target object to obtain the second voice data and the corresponding second lip movement video of the target object.

[0035] When the present invention performs digital lip training based on sound cloning, first, the first voice data of the target object, that is, the characteristic information such as the voice content and intonation emitted by the target object, is collected, and at the same time, the first lip movement video corresponding to the first voice data is obtained, and the first lip movement video records a series of movements of the lips of the target object when speaking. Then, data preprocessing is required for the first voice data of the target object and the corresponding first lip movement video.

[0036] Specifically, the first voice data of the acquired target object and the corresponding first lip movement video are subjected to data preprocessing to obtain the second voice data of the target object and the corresponding second lip movement video, including performing noise reduction processing on the first voice data of the acquired target object and the corresponding first lip movement video to obtain the second voice data of the target object and the corresponding second lip movement video. When the present invention acquires the first voice data of the target object and the corresponding first lip movement video, it may be interfered by many external factors, among which the first voice data is often mixed with various noises, and the first lip movement video may also have screen jitter, uneven light and the like. Therefore, it is necessary to perform noise reduction processing on the first voice data of the acquired target object and the corresponding first lip movement video. For the first voice data of the acquired target object, the first voice data can be subjected to noise reduction processing by an audio processing algorithm to restore clear and pure voice. For the lip movement video, a video processing algorithm is used to eliminate screen jitter, and light compensation and other means are used to improve the problem of uneven light, so as to obtain the second voice data of the target object and the corresponding second lip movement video. After the noise reduction process, the second voice data of the target object with higher quality and more in line with the requirements and the corresponding second lip movement video can be obtained.

[0037] S20: Based on the second voice data of the target object and the corresponding second lip movement video, a mapping relationship between the voice data and the lip movement of the target object is constructed. After the present invention performs data preprocessing on the first voice data and the corresponding first lip movement video of the acquired target object in step S10, feature extraction can be performed on the second voice data and the corresponding second lip movement video of the target object, and a mapping relationship between the voice data and the lip movement of the target object can be constructed based on the extracted feature information. Specifically, Figure 3 As stated, Figure 3 yes Figure 2 A specific implementation flow diagram of step S20 in the embodiment of the present invention specifically includes the following steps S21-S23:

[0038] S21: extracting acoustic features from the second speech data of the target object to obtain spectrum information of the second speech data of the target object. Specifically, key acoustic elements such as timbre, pitch, and intensity of the second speech data of the target object may be extracted through a speech processing algorithm.

[0039] S22: extract visual features from the second lip movement video to obtain lip movement information of the target object. Specifically, computer vision technology can be used to extract lip-related visual features, such as the degree of lip opening and closing, the displacement of the mouth corners, and the change curve of the lip shape.

[0040] S23: Based on the extracted frequency spectrum information and lip movement information of the target object, a mapping relationship between the speech data of the target object and the lip movement is constructed. Specifically, based on the extracted frequency spectrum information and lip movement information of the target object, a speech segment of the corresponding second speech data may be annotated to construct a mapping relationship between the speech data of the target object and the lip movement.

[0041] S30: Obtain a pre-trained multimodal speech model and text data to be cloned, and input the second speech data of the target object and the text data to be cloned into the pre-trained multimodal speech model for training to generate cloned speech data. After constructing a mapping relationship between the speech data and lip movements of the target object, the present invention inputs the second speech data of the target object and the text data to be cloned into the pre-trained multimodal speech model for training, so that the model learns the speech characteristics of the target object and can generate cloned speech data that is extremely similar to the timbre and intonation of the target object. Specifically, Figure 4 As stated, Figure 4 yes Figure 2 A specific implementation flow diagram of step S30 in the embodiment of the present invention specifically includes the following steps S31-S34:

[0042] S31: Extract features from the spectrum information of the second voice data of the target object to obtain the pitch feature information and style feature information of the second voice data. Specifically, the present invention can use a pitch detection algorithm to determine the fundamental frequency of the second voice data of the target object, and according to the value of the fundamental frequency, the pitch of the second voice data of the target object can be determined to obtain the pitch feature information of the second voice data. In addition, the style feature information of the second voice data can be extracted by analyzing the spectrum shape, frequency component changes, amplitude fluctuations and other information of the second voice data of the target object.

[0043] S32: Extract features from the acquired text data to be cloned to obtain semantic feature information and sentiment feature information of the text data to be cloned. Specifically, the present invention can use word embedding technology to extract semantic feature information of the text data to be cloned. First, each word in the text data to be cloned is converted into a corresponding word vector, and then these word vectors are combined by weighted average and other methods to form a feature vector that can characterize the text data to be cloned, so as to obtain semantic feature information of the text data to be cloned. In addition, sentiment features can be extracted using sentiment dictionaries. The sentiment dictionaries contain a large number of words with sentiment tendencies and their corresponding sentiment scores. The words in the text data to be cloned can be matched with the sentiment dictionaries, and the occurrence frequency, weight and other information of words with different sentiment tendencies in the text data to be cloned can be counted to determine the overall sentiment tendency and sentiment intensity of the text data to be cloned, so as to obtain the sentiment feature information of the text data to be cloned.

[0044] S33: Cross-modally fuse the tone feature information and style feature information of the second voice data with the semantic feature information and emotional feature information of the text data to be cloned to obtain fused target feature information. Specifically, the present invention can splice the obtained tone feature information and style feature information of the second voice data with the semantic feature information and emotional feature information of the text data to be cloned to obtain fused target feature information.

[0045] S34: Input the target feature information into a pre-trained multimodal speech model for training to generate cloned speech data. The present invention can input the fused target feature information batch by batch into a pre-trained multimodal speech model for training, and finally obtain the generated cloned speech data in the output layer of the pre-trained multimodal speech model. Specifically, Figure 5 As stated, Figure 5 yes Figure 4 A specific implementation flow diagram of step S34 in the embodiment of the present invention specifically includes the following steps S341-S343:

[0046] S341: Construct a cross-entropy loss function for the pre-trained multimodal speech model during training, and calculate the cross-entropy loss value for the pre-trained multimodal speech model during training. Specifically, the present invention can construct a cross-entropy loss function for the pre-trained multimodal speech model during training based on the probability distribution of speech-related results output by the model and the probability distribution of the corresponding real speech target, and then substitute the cloned speech data and the real speech data output by the forward propagation of the multimodal speech model after inputting the target feature information into the constructed cross-entropy loss function to calculate the cross-entropy loss value for the pre-trained multimodal speech model during training.

[0047] S342: Based on the calculated cross-entropy loss value of the pre-trained multimodal speech model during training, the various parameter information of the pre-trained multimodal speech model is updated to obtain a trained multimodal speech model. In the present invention, the gradient corresponding to each parameter of the pre-trained multimodal speech model can be calculated by a gradient optimization algorithm to determine the update direction of each parameter information of the pre-trained multimodal speech model, and then the pre-trained multimodal speech model is iteratively trained multiple times to finally obtain a trained multimodal speech model.

[0048] S343: Input the target feature information into the trained multimodal speech model to generate cloned speech data. The present invention can input the target feature information into the trained multimodal speech model in batches to predict speech data, and finally output speech data that is very similar to the timbre and intonation of the target object, thereby generating cloned speech data.

[0049] S40: Based on the mapping relationship between the constructed target object's voice data and lip movements, a digital human mouth lip movement prediction model is constructed. Specifically, after the present invention obtains the cloned voice data in step S30, the digital human mouth lip movement prediction can be performed. By constructing a digital human mouth lip movement prediction model, the digital human can automatically and accurately simulate the corresponding lip movements according to the input voice data, and predict the various action parameters that the digital human mouth lip should show according to the characteristic information of the acquired cloned voice data, thereby driving the digital human to simulate the lip movements presented when the target object outputs the cloned voice data.

[0050] S50: Input the generated cloned voice data into the digital human lip movement prediction model to predict various parameter information of the digital human lip movement. Specifically, the present invention can first extract acoustic feature information of the generated cloned voice data, such as feature information corresponding to pitch, timbre, and speech speed, and then predict the digital human's lip opening and closing degree information, lip angle displacement information, and lip shape change information based on the extracted acoustic feature information.

[0051] S60: Based on the predicted parameter information of the digital human's lip movements, drive the digital human to simulate the lip movements presented when the target object outputs the cloned voice data. Specifically, after obtaining the predicted parameter information of the digital human's lip movements, the present invention will drive the lip movement simulation according to the parameter information, so that the performance of its mouth is the same as that of a real human when speaking and making expressions. When the digital human outputs the cloned voice data, its lips can accurately make lip movements that match the voice content as the cloned voice is played, and when expressing different emotions, the corresponding expression characteristics can also be reflected through lip movements, effectively improving the fidelity and interactive experience of the digital human.

[0052] It can be seen that in the above scheme, when performing lip training for a digital human, the text data to be cloned can be first input into a pre-trained multimodal speech model for training to generate cloned speech data, and then the generated cloned speech data can be input into a digital human mouth lip movement prediction model to predict various parameter information of the digital human mouth lip movement. Finally, based on the predicted various parameter information of the digital human mouth lip movement, the digital human is driven to simulate the lip movement presented when the target object outputs the cloned speech data, thereby effectively improving the accuracy of the digital human mouth lip training.

[0053] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0054] In one embodiment, a digital mouth and lip training device based on sound cloning is provided, and the digital mouth and lip training device based on sound cloning corresponds to the digital mouth and lip training method based on sound cloning in the above embodiment. Figure 6 As shown, Figure 6 6 is a structural diagram of a digital mouth and lip training device based on sound cloning in one embodiment of the present invention, and the digital mouth and lip training device based on sound cloning includes a processing module 61, a first construction module 62, a training module 63, a second construction module 64, a prediction module 65, and a driving module 66. The functional modules are described in detail as follows:

[0055] The processing module 61 is used to obtain the first voice data of the target object and the corresponding first lip movement video, and perform data preprocessing on the obtained voice data of the target object and the corresponding lip movement video to obtain the second voice data of the target object and the corresponding second lip movement video;

[0056] A first construction module 62, configured to construct a mapping relationship between the speech data and the lip movements of the target object based on the second speech data of the target object and the corresponding second lip movement video;

[0057] The training module 63 is used to obtain a pre-trained multimodal speech model and text data to be cloned, and input the second speech data of the target object and the text data to be cloned into the pre-trained multimodal speech model for training to generate cloned speech data;

[0058] A second construction module 64 is used to construct a digital human lip movement prediction model based on the constructed mapping relationship between the speech data of the target object and the lip movement;

[0059] The prediction module 65 is used to input the generated cloned voice data into the digital mouth lip movement prediction model to predict various parameter information of the digital mouth lip movement;

[0060] The driving module 66 is used to drive the digital human to simulate the lip movements presented when the target object outputs the cloned voice data based on the various parameter information of the predicted digital human lip movements.

[0061] In one embodiment, the processing module 61 is specifically configured to:

[0062] The acquired speech data of the target object and the corresponding lip movement video are subjected to noise reduction processing to obtain second speech data of the target object and the corresponding second lip movement video.

[0063] In one embodiment, the first building module 62 is specifically configured to:

[0064] Extracting acoustic features of the second speech data of the target object to obtain frequency spectrum information of the second speech data of the target object;

[0065] Extracting visual features from the second lip movement video to obtain lip movement information of the target object;

[0066] Based on the extracted frequency spectrum information and lip movement information of the target object, a mapping relationship between the speech data and the lip movement of the target object is constructed.

[0067] In one embodiment, the training module 63 is specifically configured to:

[0068] Extracting features from the frequency spectrum information of the second speech data of the target object to obtain pitch feature information and style feature information of the second speech data;

[0069] Performing feature extraction on the acquired text data to be cloned to obtain semantic feature information and sentiment feature information of the text data to be cloned;

[0070] Cross-modally fusing the obtained tone feature information and style feature information of the second speech data with the semantic feature information and emotional feature information of the text data to be cloned to obtain fused target feature information;

[0071] The target feature information is input into a pre-trained multimodal speech model for training to generate cloned speech data.

[0072] In one embodiment, the training module 63 is specifically configured to:

[0073] Construct a cross entropy loss function for training a pre-trained multimodal speech model, and calculate the cross entropy loss value for training the pre-trained multimodal speech model;

[0074] Based on the calculated cross entropy loss value of the pre-trained multimodal speech model during training, various parameter information of the pre-trained multimodal speech model is updated to obtain a trained multimodal speech model;

[0075] The target feature information is input into a trained multimodal speech model to generate cloned speech data.

[0076] In one embodiment, the prediction module 65 is specifically configured to:

[0077] The generated cloned speech data is input into the digital human lip movement prediction model to predict the digital human's lip opening and closing degree information, lip angle displacement information and lip shape change information.

[0078] In one embodiment, the digital mouth and lip training device based on sound cloning is further used for:

[0079] Performing weighted processing on the tone feature information and style feature information of the second voice data, and the semantic feature information and emotional feature information of the text data to be cloned;

[0080] The tone feature information and style feature information of the weighted second speech data are cross-modally fused with the semantic feature information and emotional feature information of the text data to be cloned to obtain fused target feature information.

[0081] The present invention provides a digital human mouth and lip training device based on sound cloning. It can be seen that in the above scheme, when performing lip training on a digital human, the text data to be cloned can be first input into a pre-trained multimodal speech model for training to generate cloned speech data, and then the generated cloned speech data can be input into a digital human mouth and lip movement prediction model to predict various parameter information of the digital human mouth and lip movement. Finally, according to the predicted various parameter information of the digital human mouth and lip movement, the digital human is driven to simulate the lip movement presented when the target object outputs the cloned speech data, thereby effectively improving the accuracy of the digital human mouth and lip training.

[0082] For the specific definition of the digital mouth and lip training device based on sound cloning, please refer to the definition of the digital mouth and lip training method based on sound cloning above, which will not be repeated here. Each module in the above-mentioned digital mouth and lip training device based on sound cloning can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0083] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, Figure 7 1 is a schematic diagram of the structure of a computer device in one embodiment of the present invention. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of a digital lip training method based on sound cloning are implemented.

[0084] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 8 As shown, Figure 8 1 is another structural diagram of a computer device in one embodiment of the present invention. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the functions or steps on the client side of a digital lip training method based on sound cloning are implemented.

[0085] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0086] Acquire first voice data and a corresponding first lip movement video of a target object, and perform data preprocessing on the acquired voice data and the corresponding lip movement video of the target object to obtain second voice data and a corresponding second lip movement video of the target object;

[0087] Based on the second voice data of the target object and the corresponding second lip movement video, construct a mapping relationship between the voice data and the lip movement of the target object;

[0088] Acquire a pre-trained multimodal voice model and text data to be cloned, and input the second voice data of the target object and the text data to be cloned into the pre-trained multimodal voice model for training to generate cloned voice data;

[0089] Based on the mapping relationship between the speech data and lip movements of the constructed target object, a digital human lip movement prediction model is constructed;

[0090] Input the generated cloned voice data into the digital mouth lip movement prediction model to predict various parameter information of the digital mouth lip movement;

[0091] Based on the predicted parameters of the digital human's lip movements, the digital human is driven to simulate the lip movements of the target object when outputting cloned voice data.

[0092] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0093] Acquire first voice data and a corresponding first lip movement video of a target object, and perform data preprocessing on the acquired voice data and the corresponding lip movement video of the target object to obtain second voice data and a corresponding second lip movement video of the target object;

[0094] Based on the second voice data of the target object and the corresponding second lip movement video, construct a mapping relationship between the voice data and the lip movement of the target object;

[0095] Acquire a pre-trained multimodal voice model and text data to be cloned, and input the second voice data of the target object and the text data to be cloned into the pre-trained multimodal voice model for training to generate cloned voice data;

[0096] Based on the mapping relationship between the speech data and lip movements of the constructed target object, a digital human lip movement prediction model is constructed;

[0097] Input the generated cloned voice data into the digital mouth lip movement prediction model to predict various parameter information of the digital mouth lip movement;

[0098] Based on the predicted parameters of the digital human's lip movements, the digital human is driven to simulate the lip movements of the target object when outputting cloned voice data.

[0099] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0100] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0101] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0102] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A digital mouth lip training method based on sound cloning, characterized in that: include: Acquire first voice data and a corresponding first lip movement video of a target object, and perform data preprocessing on the acquired first voice data and the corresponding first lip movement video of the target object to obtain second voice data and a corresponding second lip movement video of the target object; Based on the second voice data of the target object and the corresponding second lip movement video, construct a mapping relationship between the voice data and the lip movement of the target object; Acquire a pre-trained multimodal voice model and text data to be cloned, and input the second voice data of the target object and the text data to be cloned into the pre-trained multimodal voice model for training to generate cloned voice data; Based on the mapping relationship between the speech data and lip movements of the constructed target object, a digital human lip movement prediction model is constructed; Input the generated cloned voice data into the digital mouth lip movement prediction model to predict various parameter information of the digital mouth lip movement; Based on the predicted parameters of the digital human's lip movements, the digital human is driven to simulate the lip movements of the target object when outputting cloned voice data.

2. The digital mouth and lip training method based on sound cloning according to claim 1, characterized in that: The first speech data of the target object and the corresponding first lip movement video are preprocessed to obtain the second speech data of the target object and the corresponding second lip movement video, including The acquired first voice data of the target object and the corresponding first lip movement video are subjected to noise reduction processing to obtain the second voice data of the target object and the corresponding second lip movement video.

3. The digital mouth and lip training method based on sound cloning according to claim 1, characterized in that: Construct a mapping relationship between the target object's speech data and lip movements, including: Extracting acoustic features of the second speech data of the target object to obtain frequency spectrum information of the second speech data of the target object; Extracting visual features from the second lip movement video to obtain lip movement information of the target object; Based on the extracted frequency spectrum information and lip movement information of the target object, a mapping relationship between the speech data and the lip movement of the target object is constructed.

4. The digital mouth and lip training method based on sound cloning according to claim 3 is characterized in that: The method of inputting the target object's voice data and the text data to be cloned into a pre-trained multimodal voice model for training to generate cloned voice data includes: Extracting features from the frequency spectrum information of the second speech data of the target object to obtain pitch feature information and style feature information of the second speech data; Performing feature extraction on the acquired text data to be cloned to obtain semantic feature information and sentiment feature information of the text data to be cloned; Cross-modally fusing the tone feature information and style feature information of the second speech data with the semantic feature information and emotional feature information of the text data to be cloned to obtain fused target feature information; The target feature information is input into a pre-trained multimodal speech model for training to generate cloned speech data.

5. The digital mouth and lip training method based on sound cloning according to claim 4 is characterized in that: The step of inputting the target feature information into a pre-trained multimodal speech model for training to generate cloned speech data includes: Construct a cross entropy loss function for training a pre-trained multimodal speech model, and calculate the cross entropy loss value for training the pre-trained multimodal speech model; Based on the calculated cross entropy loss value of the pre-trained multimodal speech model during training, various parameter information of the pre-trained multimodal speech model is updated to obtain a trained multimodal speech model; The target feature information is input into a trained multimodal speech model to generate cloned speech data.

6. The digital mouth and lip training method based on sound cloning according to claim 1 is characterized in that: The generated cloned voice data is input into the digital mouth lip movement prediction model to predict various parameter information of the digital mouth lip movement, including: The generated cloned voice data is input into the digital human lip movement prediction model to predict the digital human's lip opening and closing degree information, lip angle displacement information and lip shape change information.

7. The digital mouth and lip training method based on sound cloning according to claim 4 is characterized in that: The method further comprises: Performing weighted processing on the tone feature information and style feature information of the second voice data, and the semantic feature information and emotional feature information of the text data to be cloned; The tone feature information and style feature information of the weighted second speech data are cross-modally fused with the semantic feature information and emotional feature information of the text data to be cloned to obtain fused target feature information.

8. A digital mouth and lip training device based on sound cloning, characterized in that: The device is used to implement the digital mouth and lip training method based on sound cloning according to any one of claims 1 to 7, comprising: a processing module, configured to obtain first voice data and a corresponding first lip movement video of a target object, and perform data preprocessing on the obtained first voice data and the corresponding first lip movement video of the target object to obtain second voice data and a corresponding second lip movement video of the target object; A first construction module is used to construct a mapping relationship between the voice data and the lip movement of the target object based on the second voice data of the target object and the corresponding second lip movement video; A training module, used to obtain a pre-trained multimodal voice model and text data to be cloned, and input the second voice data of the target object and the text data to be cloned into the pre-trained multimodal voice model for training to generate cloned voice data; The second construction module is used to construct a digital human lip movement prediction model based on the mapping relationship between the constructed target object's speech data and lip movements; A prediction module, used to input the generated cloned voice data into a digital human mouth lip movement prediction model to predict various parameter information of the digital human mouth lip movement; The driving module is used to drive the lip movements presented by the digital human to simulate the target object when outputting cloned voice data based on the various parameter information of the predicted digital human lip movements.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the digital lip training method based on sound cloning according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the digital lip training method based on sound cloning according to any one of claims 1 to 7 are implemented.

Images