Model training method, face video generation method, device, equipment and medium
By calculating the fusion of the matching probability of input data and training annotation results and the initial loss value, dynamically adjusting the loss value, the problem of pre-trained models performing poorly on unstable data sets is solved, and the accuracy and efficiency of model training are improved.
Patent Information
- Application Number
- CN202110246125.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-05
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, when using a pre-trained model as a loss function, the model performs unstable on some data sets, resulting in a large difference between the calculated loss value and the real value, affecting the accuracy and efficiency of model training.
By calculating the matching probability of the input data and its corresponding training annotation results and the initial loss value, the dynamically adjusted target loss value is obtained. The adaptive dynamic loss function is used to supervise model training to avoid expert models on unstable data sets Negative effects.
It improves the accuracy and speed of model training, enhances hardware execution efficiency, avoids training misleading caused by unstable expert models, and improves the convergence effect of the overall model.
Smart Images

Figure CN112966607B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence, in particular to computer vision and deep learning technology, and specifically to a model training method, a face video generation method, an apparatus, a device, a medium and a program product. Background Art
[0002] With the development of theory and computing power, deep learning algorithms have become the most important technology in the field of artificial intelligence. In the process of training deep learning models, loss functions play a vital role. A good loss function often needs to be able to correctly measure the quality of the current model and guide the model to converge in the direction of positive benefits during the training process.
[0003] In the prior art, a common method is to use a trained model as an "expert" to supervise the training of another model, and use this expert model as one of the loss functions. This method has achieved good results in many tasks. However, on some data sets, the performance of these "expert" models is not particularly stable, so it is impossible to calculate the correct loss value. In fact, the final calculated loss value is very different from the actual value, which is harmful to the training of the model. Summary of the invention
[0004] The present application provides a model training method, a face video generation method, an apparatus, a device, a medium and a program product to improve the accuracy of model training.
[0005] In a first aspect, the present application provides a model training method, comprising:
[0006] Inputting input data for training a main model into the main model to obtain output data of the main model;
[0007] Using a pre-trained expert model, an initial loss value is calculated based on the input data and the output data;
[0008] Using the expert model, calculating the matching probability of the input data and its corresponding training annotation results;
[0009] The matching probability is fused with the initial loss value to obtain a target loss value, wherein the target loss value is used for supervised training of the main model.
[0010] In a second aspect, the present application also provides a method for generating a face video based on speech drive, comprising:
[0011] Outputting a facial lip shape sequence according to the input facial image and voice data using a pre-trained voice-driven lip shape model, wherein the voice-driven lip shape model is trained according to the model training method described in any embodiment of the present application;
[0012] Based on the facial lip shape sequence, a facial video driven by the voice data is generated on the facial image.
[0013] In a third aspect, the present application also provides a model training device, comprising:
[0014] An output data determination module, used for inputting input data for training a main model into the main model to obtain output data of the main model;
[0015] An initial loss value determination module, used to calculate an initial loss value according to the input data and the output data using a pre-trained expert model;
[0016] A matching probability calculation module, used to calculate the matching probability of the input data and its corresponding training annotation results using the expert model;
[0017] A target loss value determination module is used to fuse the matching probability with the initial loss value to obtain a target loss value, wherein the target loss value is used for supervised training of the main model.
[0018] In a fourth aspect, the present application also provides a voice-driven face video generation device, comprising:
[0019] A face lip shape sequence determination module is used to obtain a face lip shape sequence according to an input face image and voice data using a pre-trained voice-driven lip shape model, wherein the voice-driven lip shape model is trained by a model training device;
[0020] A face video generation module is used to generate a face video driven by the voice data on the face image based on the face lip shape sequence.
[0021] In a fifth aspect, the present application further provides an electronic device, including:
[0022] at least one processor; and
[0023] a memory communicatively connected to the at least one processor; wherein,
[0024] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the model training method described in any embodiment of the present application.
[0025] In a sixth aspect, the present application also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the model training method described in any embodiment of the present application.
[0026] In the seventh aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the model training method described in any embodiment of the present application.
[0027] In an eighth aspect, the present application further provides an electronic device, including:
[0028] at least one processor; and
[0029] a memory communicatively connected to the at least one processor; wherein,
[0030] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the voice-driven facial video generation method described in any embodiment of the present application.
[0031] In a ninth aspect, the present application also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the voice-driven facial video generation method described in any embodiment of the present application.
[0032] In the tenth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the voice-driven facial video generation method described in any embodiment of the present application.
[0033] It should be understood that the content described in this section is not intended to identify the key or important features of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easy to understand through the following description, and other effects of the above optional methods will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.
[0035] Figure 1 is a flow chart of a model training method according to an embodiment of the present application;
[0036] Figure 2 is a flow chart of a model training method according to an embodiment of the present application;
[0037] Figure 3 is a flow chart of a model training method according to an embodiment of the present application;
[0038] Figure 4 is a flow chart of a method for generating a face video based on voice-driven operation according to an embodiment of the present application;
[0039] Figure 5 is a structural schematic diagram of a model training device according to an embodiment of the present application;
[0040] Figure 6 is a structural schematic diagram of a voice-driven face video generation device according to an embodiment of the present application;
[0041] Figure 7 It is a block diagram of an electronic device used to implement the model training method of an embodiment of the present application. DETAILED DESCRIPTION
[0042] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0043] Figure 1 1 is a flow chart of a model training method according to an embodiment of the present application. This embodiment can be applied to supervised training of a main model using an expert model, and involves computer vision and deep learning technology. The method can be executed by a model training device, which is implemented in software and / or hardware, and is preferably configured in an electronic device, such as a server or a computer device. Figure 1 As shown, the method specifically includes the following:
[0044] S101, inputting the input data for training the main model into the main model to obtain the output data of the main model.
[0045] Among them, the main model is the model to be trained, such as a voice-driven lip shape model or other models that perform any deep learning tasks. The model obtains output data based on input data.
[0046] S102. Calculate an initial loss value based on input data and output data using a pre-trained expert model.
[0047] Among them, the expert model is a pre-trained model, such as a lip synchronization model. After the expert model is trained, the parameters of the expert model will be fixed, and it will be switched to the test mode. The input data and output data of the main model will be input into the expert model, and the expert model will be used to calculate the matching degree between the input data and the output data of the main model. Then, the initial loss value is determined according to the matching degree. For example, the expert model is used as one of the loss functions, and the initial loss value is calculated according to the matching degree and the loss function. The training method of the expert model and the method for calculating the loss value can be selected from any of the existing technologies, which will not be repeated here.
[0048] S103: Using the expert model, calculate the matching probability of the input data and its corresponding training annotation results.
[0049] The training annotation result is the real annotation result corresponding to the input data for training. The input data and the training annotation result are input into the expert model, and the expert model calculates the matching probability between the input data and the training annotation result.
[0050] Since the training annotation results are the real results corresponding to the input data, and the input data and the training annotation results are completely matched, theoretically, the matching probability calculated by the expert model should be equal to 1.0, but in fact, the expert model is not stable on some data sets. Therefore, if the matching probability is closer to 1.0, it means that the expert model's judgment on the input data pair is more accurate, and at this time, the "expert" can be trusted more; conversely, if the matching probability is further away from 1.0, it means that the expert model's judgment is less accurate and less trustworthy. At this time, if the expert model is still used to calculate the loss value and supervise the training of the main model, it will affect the training effect and the accuracy of the model training.
[0051] S104: Fusing the matching probability with the initial loss value to obtain a target loss value, wherein the target loss value is used for supervised training of the main model.
[0052] In the embodiment of the present application, by allowing the expert model to calculate the matching probability of the input data and the training annotation results, the matching probability can be regarded as a confidence level to help determine how credible the expert model is, thereby using the matching probability to dynamically adjust the initial loss value, making the loss function an adaptive dynamic function, and the target loss value calculated after fusion is also a dynamic adjustment value, thereby avoiding the problem of affecting the training effect of the main model when the expert model performs poorly on certain data sets. Therefore, the present application can also accelerate the convergence of the model and increase the training speed of the model, thereby improving the overall hardware execution efficiency and saving hardware resources.
[0053] The technical solution of the embodiment of the present application, after calculating the initial loss value, uses the expert model to calculate the matching probability of the input data and its corresponding training annotation results, thereby fusing the matching probability with the initial loss value to obtain a dynamically adjusted target loss value, and then, by introducing an adaptive dynamic loss function, the training of the main model can be better guided. Even if the expert model or loss function is not good enough, the final loss value can be adaptively adjusted according to the current situation to improve the effect and accuracy of model training.
[0054] Figure 2 1 is a flow chart of a model training method according to an embodiment of the present application. This embodiment is further optimized on the basis of the above embodiment. Figure 2 As shown, the method specifically includes the following:
[0055] S201, inputting the input data for training the main model into the main model to obtain the output data of the main model.
[0056] S202: Calculate the matching degree between input data and output data using the expert model.
[0057] S203: Calculate an initial loss value according to the matching degree and a pre-configured loss function.
[0058] S204: Calculate the matching probability of the input data and its corresponding training annotation results using the expert model.
[0059] S205: Determine whether the matching probability is higher than a preset threshold. If yes, execute S206; otherwise, execute S207.
[0060] S206. Multiply the initial loss value by the matching probability as a weight to obtain a target loss value, wherein the target loss value is used for supervised training of the main model.
[0061] S207: Set the matching probability to zero and do not calculate the current target loss value.
[0062] The threshold can be preset as needed. For example, the threshold can be set to 0.5. That is, when the matching probability is higher than 0.5, it indicates that the expert model performs well and is trustworthy. If it is not higher than 0.5, it indicates that the expert model performs poorly on the current data set and is not trustworthy. This means that the confidence of the expert model is too low and may even lead the main model to iterate in a bad direction. Therefore, when the matching probability is not higher than the threshold, the matching probability is set to zero and the current target loss value is not calculated to avoid affecting the training of the main model.
[0063] When the matching probability is higher than the threshold, the matching probability can be used as a weight to multiply the initial loss value, so that the calculated target loss value is a dynamically adjusted value, and it is a value dynamically adjusted according to the confidence of the expert model. Therefore, the calculated target loss value is more accurate, and it is more accurate to use the target loss value to supervise the training of the main model.
[0064] The technical solution of the embodiment of the present application uses a threshold to determine whether the confidence of the expert model is too low, thereby preventing the expert model from guiding the main model to iterate in a bad direction, thereby affecting the training of the main model and improving the training effect of the main model.
[0065] Figure 3 2 is a flow chart of a model training method according to an embodiment of the present application. In this embodiment, the voice-driven lip shape task is taken as an example, and further optimization is performed on the basis of the above embodiment. Figure 3 As shown, the method specifically includes the following:
[0066] S301, inputting the face image and voice data used for training the voice-driven lip shape model into the voice-driven lip shape model, and obtaining the face lip shape sequence output by the voice-driven lip shape model.
[0067] In this embodiment, the speech-driven lip shape model is the main model to be trained, and the input data for training includes facial images and speech data. The speech-driven lip shape model outputs a facial lip shape sequence based on the input data. Based on the facial lip shape sequence, a facial video driven by the input speech data can be generated on the facial image.
[0068] S302: Using the pre-trained lip synchronization model as an expert model, and using the expert model to calculate the matching degree between the speech data and the facial lip shape sequence.
[0069] Among them, the expert model is a lip synchronization model, which is used to calculate the matching degree between the input speech data and the facial lip shape sequence.
[0070] S303: Calculate an initial loss value according to the matching degree and a pre-configured loss function.
[0071] For example, the expert model is used as one of the loss functions, and the initial loss value is calculated according to the matching degree and the loss function. The loss function can be configured according to the actual situation.
[0072] S304: Calculate the matching probability of the speech data and its corresponding training annotation results using the expert model.
[0073] S305, determine whether the matching probability is higher than a preset threshold, if so, execute S306 and then execute S307, otherwise execute S308.
[0074] S306: Multiply the initial loss value by the matching probability as a weight to obtain a target loss value.
[0075] S307. Perform supervised training on the speech-driven lip shape model using the target loss value.
[0076] S308: Set the matching probability to zero and do not calculate the target loss value of this round.
[0077] Among them, in order to measure the credibility of the lip synchronization model as an expert model, the speech data and the annotated true value training annotation results are input into the expert model to calculate the matching probability of the speech data and the training annotation results. If the matching probability is closer to 1.0, it means that the lip synchronization model judges the input data more accurately, and the "expert" can be trusted more. On the contrary, if the matching probability is farther away from 1.0, it means that the lip synchronization model judges less accurately and is less trustworthy. Therefore, the matching probability can be regarded as the confidence of the lip synchronization model.
[0078] Moreover, in order to avoid the lip synchronization model's confidence being too low, which may even lead the speech-driven lip shape model to iterate in a bad direction, the present embodiment pre-sets a threshold. When the matching probability is not higher than the threshold, the matching probability is set to zero, and the current target loss value is not calculated to avoid affecting the speech-driven lip shape model training. When the matching probability is higher than the threshold, the matching probability is used as a weight to multiply the initial loss value to obtain the target loss value, thereby introducing dynamic loss according to the confidence of the lip synchronization model, and using the dynamic target loss value to perform supervised training on the speech-driven lip shape model, thereby improving the accuracy of model training and the prediction accuracy of the speech-driven lip shape model.
[0079] The technical solution of the embodiment of the present application, by introducing an adaptive dynamic loss function and dynamically adjusting the loss value, can improve the accuracy of speech-driven lip shape model training regardless of whether the lip synchronization model performs well enough, and can better guide the training of the model. Even in the case of the so-called "expert" model or the loss function is not good enough, the final loss value can be adaptively adjusted according to the current situation. Moreover, the technical solution of the embodiment of the present application, in addition to being applied to speech-driven lip shape tasks, can also be extended to many different deep learning tasks, and has strong portability.
[0080] Figure 4This is a flow chart of a method for generating a face video based on voice-driven according to an embodiment of the present application. This embodiment can be applied to the case where an expert model is used to supervise the training of a voice-driven lip shape model, and the trained voice-driven lip shape model is used to generate a voice-driven face video, which involves computer vision and deep learning technology. The method can be performed by a voice-driven face video generation device, which is implemented in software and / or hardware, and is preferably configured in an electronic device, such as a server or a computer device. Figure 4 As shown, the method specifically includes the following:
[0081] S401. Utilize a pre-trained speech-driven lip shape model to output a facial lip shape sequence according to an input facial image and speech data, wherein the speech-driven lip shape model is trained according to the model training method described in any embodiment of the present application.
[0082] S402: Generate a face video driven by speech data on the face image based on the face lip shape sequence.
[0083] According to the model training method described in any embodiment of the present application, by introducing an adaptive dynamic loss function and dynamically adjusting the loss value, the accuracy of the speech-driven lip shape model training can be improved regardless of whether the lip synchronization model performance is good enough, and the training of the speech-driven lip shape model can be better guided. Even in the case of the so-called "expert" model or the loss function is not good enough, the final loss value can be adaptively adjusted according to the current situation. Therefore, the speech-driven lip shape model trained by the model training method described in any embodiment of the present application has better performance in the speech-driven lip shape task, and can more accurately predict the facial lip shape sequence based on the input facial image and voice data, thereby obtaining a more accurate voice data-driven facial video.
[0084] Figure 5 Schematic diagram of the structure of the model training device according to the embodiment of the present application. The present embodiment can be applied to the situation where the model is supervised and trained using an expert model, involving computer vision and deep learning technology. The device can implement the model training method described in any embodiment of the present application. Figure 5 As shown, the device 500 specifically includes:
[0085] An output data determination module 501 is used to input the input data used to train the main model into the main model to obtain output data of the main model;
[0086] An initial loss value determination module 502, used to calculate an initial loss value according to the input data and the output data using a pre-trained expert model;
[0087] A matching probability calculation module 503 is used to calculate the matching probability of the input data and its corresponding training annotation results using the expert model;
[0088] The target loss value determination module 504 is used to fuse the matching probability with the initial loss value to obtain a target loss value, wherein the target loss value is used to perform supervised training on the main model.
[0089] Optionally, the initial loss value determination module 502 includes:
[0090] A matching degree calculation unit, used to calculate the matching degree between the input data and the output data using the expert model;
[0091] An initial loss value calculation unit is used to calculate the initial loss value according to the matching degree and a preconfigured loss function.
[0092] Optionally, the target loss value determination module 504 is specifically used to:
[0093] The matching probability is used as a weight and multiplied by the initial loss value to obtain the target loss value.
[0094] Optionally, the target loss value determination module 504 includes:
[0095] A first determining unit, configured to, if the matching probability is higher than a preset threshold, multiply the initial loss value by the matching probability as a weight to obtain the target loss value;
[0096] The second determining unit is configured to set the matching probability to zero if the matching probability is not higher than the threshold.
[0097] Optionally, the input data includes facial images and speech data; the main model is a speech-driven lip shape model; and the output data includes a facial lip shape sequence.
[0098] Optionally, the expert model is a lip synchronization model, which is used to calculate the matching degree between speech data and facial lip shape sequences.
[0099] The model training device 500 provided in the embodiment of the present application can execute the model training method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method. The contents not described in detail in this embodiment can refer to the description in any method embodiment of the present application.
[0100] Figure 6This is a schematic diagram of the structure of a voice-driven face video generation device according to an embodiment of the present application. This embodiment can be applied to the situation where an expert model is used to supervise the training of a voice-driven lip shape model, and the trained voice-driven lip shape model is used to generate a voice-driven face video, involving computer vision and deep learning technology. The device can implement the voice-driven face video generation method described in any embodiment of the present application. Figure 6 As shown, the device 600 specifically includes:
[0101] A face lip shape sequence determination module 601 is used to obtain a face lip shape sequence according to an input face image and voice data using a pre-trained voice-driven lip shape model, wherein the voice-driven lip shape model is trained by the model training device described in any embodiment of the present application;
[0102] The face video generation module 602 is used to generate a face video driven by speech data on a face image based on a face lip shape sequence.
[0103] The voice-driven face video generation device 600 provided in the embodiment of the present application can execute the voice-driven face video generation method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method. For the contents not described in detail in this embodiment, please refer to the description in any method embodiment of the present application.
[0104] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.
[0105] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0106] like Figure 7As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0107] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0108] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the model training method. For example, in some embodiments, the model training method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the model training method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the model training method in any other appropriate manner (e.g., by means of firmware).
[0109] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0110] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0111] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0113] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0114] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services. The server may also be a server of a distributed system, or a server combined with a blockchain.
[0115] In addition, according to an embodiment of the present application, the present application also provides another electronic device, another readable storage medium and another computer program product, which are used to execute one or more steps of the voice-driven face video generation method described in any embodiment of the present application. The specific structure and program code can be found in Figure 7 The description of the contents of the illustrated embodiment will not be repeated here.
[0116] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0117] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A model training method, comprising: Inputting input data for training a main model into the main model to obtain output data of the main model; The input data includes face images and voice data; The main model is a speech-driven lip shape model; the output data includes a human face lip shape sequence; Using a pre-trained expert model, an initial loss value is calculated based on the input data and the output data; Using the expert model, calculating the matching probability of the input data and its corresponding training annotation results; The expert model is a lip synchronization model, which is used to calculate the matching degree between speech data and human face lip shape sequence; The matching probability is fused with the initial loss value, specifically, the matching probability is used as a weight and multiplied by the initial loss value to obtain a target loss value, wherein the target loss value is used for supervised training of the main model.
2. The method according to claim 1, wherein: The method of using the pre-trained expert model to calculate the initial loss value according to the input data and the output data includes: Calculating the matching degree between the input data and the output data using the expert model; The initial loss value is calculated according to the matching degree and a preconfigured loss function.
3. The method according to claim 1, wherein: The step of multiplying the initial loss value by the matching probability as a weight to obtain the target loss value includes: If the matching probability is higher than a preset threshold, the matching probability is used as a weight and multiplied by the initial loss value to obtain the target loss value; If the matching probability is not higher than the threshold, the matching probability is set to zero.
4. A method for generating face videos based on speech drive, comprising: Outputting a facial lip shape sequence according to an input facial image and voice data using a pre-trained voice-driven lip shape model, wherein the voice-driven lip shape model is trained according to the model training method according to any one of claims 1 to 3; Based on the facial lip shape sequence, a facial video driven by the voice data is generated on the facial image.
5. A model training device, comprising: An output data determination module, used for inputting input data for training a main model into the main model to obtain output data of the main model; The input data includes face images and voice data; The main model is a speech-driven lip shape model; the output data includes a human face lip shape sequence; An initial loss value determination module, used to calculate an initial loss value according to the input data and the output data using a pre-trained expert model; A matching probability calculation module, used to calculate the matching probability of the input data and its corresponding training annotation results using the expert model; The expert model is a lip synchronization model, which is used to calculate the matching degree between speech data and human face lip shape sequence; The target loss value determination module is used to fuse the matching probability with the initial loss value, specifically to multiply the matching probability with the initial loss value as a weight to obtain a target loss value, wherein the target loss value is used for supervised training of the main model.
6. The device according to claim 5, wherein: The initial loss value determination module comprises: A matching degree calculation unit, used to calculate the matching degree between the input data and the output data using the expert model; An initial loss value calculation unit is used to calculate the initial loss value according to the matching degree and a preconfigured loss function.
7. The device according to claim 5, wherein: The target loss value determination module comprises: A first determining unit, configured to, if the matching probability is higher than a preset threshold, multiply the initial loss value by the matching probability as a weight to obtain the target loss value; The second determining unit is configured to set the matching probability to zero if the matching probability is not higher than the threshold.
8. A voice-driven face video generation device, comprising: A face lip shape sequence determination module, used to obtain a face lip shape sequence according to an input face image and voice data using a pre-trained voice-driven lip shape model, wherein the voice-driven lip shape model is trained by a model training device according to any one of claims 5 to 7; A face video generation module is used to generate a face video driven by the voice data on the face image based on the face lip shape sequence.
9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the model training method described in any one of claims 1-3.
10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the model training method described in any one of claims 1 to 3.
11. A computer program product, comprising a computer program, which, when executed by a processor, implements the model training method according to any one of claims 1 to 3.
12. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the voice-driven facial video generation method described in claim 4.
13. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the voice-driven facial video generation method described in claim 4.
14. A computer program product, comprising a computer program, which, when executed by a processor, implements the method for generating human face videos based on speech-driven operation according to claim 4.
