Virtual digital population synchronization method, system, device, medium and program product

Abstract

The invention provides a virtual digital population type synchronization method, system and device, a medium and a program product, and the method comprises the following steps: analyzing an input voice signal, and obtaining a voice feature; taking the voice features as input of a lip movement prediction model, and predicting corresponding lip movement parameters; driving a face model of the virtual digital human to perform real-time rendering according to the lip movement parameters, and realizing fine movement control of lips; and obtaining user feedback data, and continuously optimizing and adjusting the lip movement prediction model based on the user feedback data. Through advanced technologies such as deep learning, voice analysis and lip movement prediction, accurate synchronization of the virtual digital population type and the voice content is realized, and the expression ability of the virtual character and the user interaction experience are improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer graphics, artificial intelligence and multimedia interactive technology, and in particular to a virtual digital population synchronization method, system, equipment, medium and program product. Background Art

[0002] With the rapid development of computer technology, virtual digital humans have become important roles in many fields such as digital entertainment, online education, and customer service. However, when virtual digital humans in the prior art express speech, the synchronization between their lip shapes and speech content often lags, does not match, or is too mechanical, which seriously affects the authenticity and immersion of the user experience. Therefore, it is urgent to develop a virtual digital human lip shape synchronization method that can accurately match speech content and achieve natural and smooth lip shape changes. Summary of the invention

[0003] In order to achieve the above-mentioned purpose and other advantages of the present invention, the first purpose of the present invention is to provide a virtual digital population type synchronization method, comprising the following steps: Analyze the input speech signal to obtain speech features; Using the speech features as input of a lip movement prediction model to predict corresponding lip movement parameters; Driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters, and realizing fine lip movement control; Obtain user feedback data, and continuously optimize and adjust the lip movement prediction model based on the user feedback data.

[0004] Furthermore, the step of analyzing the input voice signal to obtain voice features includes: Receiving an input voice signal and preprocessing the voice signal; The speech signal is converted into a text sequence using speech recognition technology, and speech features are extracted.

[0005] Furthermore, the lip movement prediction model is constructed based on a deep learning framework.

[0006] Furthermore, the training of the lip movement prediction model includes: Use the labeled speech-lip movement data to train the model and learn the mapping relationship between speech features and lip movement parameters.

[0007] Furthermore, the step of achieving fine lip movement control includes: Use skeletal animation or shape key technology to achieve fine movement control of lips.

[0008] Furthermore, the method further comprises the steps of: Realize lip shape changes under various expressions and emotions.

[0009] Furthermore, the step of obtaining user feedback data includes: Introduce a user feedback mechanism to collect user evaluation data on the synchronization effect of virtual digital population.

[0010] Furthermore, the step of driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters also includes: Preprocess the input video; The preprocessed video and the speech signal are input into a lip synchronization network to generate a lip synchronization video.

[0011] Furthermore, the step of preprocessing the input video includes: Cut out the person from the video frame, use the semantic-guided reproduction network to modify each frame of the person's expression in the video frame according to the expression template, and generate a video with standardized expressions; Mask the mouth part in the original image; The video with the standard expression is aligned and connected with the mask processing result.

[0012] Furthermore, the step of inputting the preprocessed video and the speech signal into a lip synchronization network to generate a lip synchronization video includes: The alignment and connection results and the speech signal are input into the lip synchronization network, and the best mouth shape position in the audio signal and the video frame is matched through a deep learning model to achieve lip synchronization.

[0013] Furthermore, the step of driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters also includes: The lip-synced video is post-processed through an identity-aware face augmentation network to improve the photorealism of the synthesized face.

[0014] Furthermore, the step of driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters also includes: The lip-sync video is processed by super-resolution reconstruction technology to improve the video quality.

[0015] Furthermore, the step of driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters also includes: Reduce processing time by optimizing algorithms to ensure real-time lip sync; Post-processing sharpening technology is used to eliminate data domain offset and improve the high fidelity of the output video.

[0016] The second object of the present invention is to provide a virtual digital population type synchronization system, which applies the above method and includes a speech analysis module, a lip movement prediction module, a lip movement rendering module, and a feedback and optimization module; wherein, The speech analysis module is used to analyze the input speech signal to obtain speech features; The lip movement prediction module is used to use the speech features as input of a lip movement prediction model to predict corresponding lip movement parameters; The lip movement rendering module is used to drive the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters, and to achieve fine movement control of the lips; The feedback and optimization module is used to obtain user feedback data and continuously optimize and adjust the lip movement prediction model based on the user feedback data.

[0017] A third object of the present invention is to provide a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0018] A fourth object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0019] A fifth object of the present invention is to provide a computer program product, comprising a computer program, and implementing the steps of the above method when the computer program is executed by a processor.

[0020] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a method, system, device, medium and program product for synchronizing the type of a virtual digital human. The method comprises the following steps: analyzing an input voice signal to obtain voice features; using the voice features as input to a lip movement prediction model to predict corresponding lip movement parameters; driving the facial model of a virtual digital person to render in real time according to the lip movement parameters, and realizing fine lip movement control; obtaining user feedback data, and continuously optimizing and adjusting the lip movement prediction model based on the user feedback data. The present invention achieves accurate synchronization of the type of a virtual digital person and voice content through advanced technologies such as deep learning, voice analysis and lip movement prediction, thereby improving the expressiveness of virtual characters and user interaction experience.

[0021] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. The specific implementation of the present invention is given in detail by the following embodiments and their accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 This is a flow chart of the virtual digital population synchronization method of Example 1; Figure 2 This is a flow chart of speech signal analysis of Example 1; Figure 3 The lip-sync video generation process of Example 1 Figure 1 ; Figure 4 This is a flow chart of input video preprocessing in Example 1; Figure 5 This is a schematic diagram of the model framework of Example 1; Figure 6 The lip-sync video generation process of Example 1 Figure 2 ; Figure 7 It is a schematic diagram of a virtual digital population synchronization system of Example 2; Figure 8 This is a schematic diagram of a computer device in Example 3; Fig. 9 This is a schematic diagram of a computer-readable storage medium of Example 4. DETAILED DESCRIPTION

[0023] The present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. It should be noted that, under the premise of no conflict, the embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0024] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0025] The figure numbers in this application are only used to distinguish the various steps in the scheme, and are not used to limit the execution order of the various steps. The specific execution order is subject to the description in the specification.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0027] Example 1 A virtual digital population type synchronization method is used to achieve high synchronization between the virtual digital population type and the voice content, aiming to enhance the realism and naturalness of the virtual character in the communication process, and is widely used in multiple scenarios such as virtual anchors, game characters, educational simulations, and film and television production. Figure 1 As shown, the method comprises the following steps: S1. Analyze the input speech signal to obtain speech features; In some embodiments, Figure 2 As shown, the step of analyzing the input voice signal to obtain voice features includes: S11, receiving an input voice signal, and preprocessing the voice signal (such as denoising, framing, etc.); S12. Using speech recognition technology (such as automatic speech recognition ASR) to convert the speech signal into a text sequence and extract speech features.

[0028] S2, using the speech feature as input of a lip movement prediction model to predict corresponding lip movement parameters; Preferably, the lip movement prediction model is constructed based on a deep learning framework (such as convolutional neural network CNN, recurrent neural network RNN ​​or Transformer, etc.).

[0029] The training of the lip movement prediction model includes: The model is trained using a large amount of labeled speech-lip movement data to learn the mapping relationship between speech features and lip movement parameters (such as lip shape, degree of opening and closing, movement trajectory, etc.).

[0030] In the real-time prediction stage, the trained lip movement prediction model outputs the corresponding lip movement parameter sequence based on the input speech features.

[0031] S3, driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters, and realizing fine movement control of the lips; In some embodiments, the step of achieving fine lip motion control includes: Use skeletal animation or shape keys to achieve fine lip movement control and ensure accurate synchronization between lip shape and speech content.

[0032] In some embodiments, the steps include: Realize lip shape changes under various expressions and emotions, that is, support lip shape changes under various expressions and emotions, and enhance the expressiveness and emotional transmission ability of virtual characters.

[0033] In some embodiments, Figure 3 , Figure 5 , Figure 6As shown, the step of driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters also includes: S31, preprocessing the input video; Specifically, Figure 4 As shown, the step of preprocessing the input video includes: S311. Cut out a person from the video frame and neutralize the person's expression (i.e., "closed mouth" state). Specifically, a semantically guided reproduction network is used to modify each frame of the person's expression in the video frame according to an expression template to generate a video with standardized expressions. This can eliminate expression changes in the original video and make the generated lip shape more consistent.

[0034] This embodiment introduces a semantically guided reproduction network to normalize the expressions in the video frames, eliminating the impact of expression changes in the original video on lip synchronization, making the generated lip shapes more consistent and accurate.

[0035] S312, masking the mouth part in the original image to facilitate subsequent lip synchronization; S313: aligning and connecting the video with the standard expression and the mask processing result.

[0036] S32: Input the preprocessed video and the speech signal into a lip synchronization network to generate a lip synchronization video.

[0037] Specifically, the step of inputting the preprocessed video and the speech signal into a lip synchronization network to generate a lip synchronization video includes: The alignment and connection results and the speech signal are input into the lip synchronization network, and the deep learning model is used to match the audio signal with the optimal mouth shape position in the video frame to achieve lip synchronization. This may involve converting the speech signal into a spectrogram, and then searching and matching the corresponding lip shape in the video frame.

[0038] The lip synchronization network achieves accurate matching of speech signals with mouth shapes in video frames, improving the accuracy and fidelity of lip synchronization.

[0039] In some embodiments, the step of driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters further includes: The lip-sync video is post-processed through an identity-aware face augmentation network to improve the photorealism of the synthesized face. In particular, the lower half of the face (such as the teeth) is optimized to reduce artifacts and synthesis bounding box issues.

[0040] This embodiment uses an identity-aware facial enhancement network to post-process the synchronized video, especially optimizing the lower half of the face (such as the teeth), which significantly improves the photo realism of the synthesized face.

[0041] In order to obtain a higher resolution output, in some embodiments, the step of driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters further includes: The lip-sync video is processed by super-resolution reconstruction technology (such as GFP-GAN, GPEN, etc.) to further improve the video quality.

[0042] In some embodiments, the step of driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters further includes: For real-time applications, the processing time can be reduced by optimizing the algorithm to ensure the real-time performance of lip synchronization; Post-processing sharpening techniques (e.g., domain adaptive post-processing networks) are used to eliminate data domain offset and improve the fidelity of the output video.

[0043] This embodiment introduces a domain adaptive post-processing network, which eliminates data domain offset and improves the high fidelity of the output video by learning the migration / adaptation from the big data distribution domain to the single-person data distribution domain.

[0044] This embodiment proposes a variational motion generator, which solves the problem of poor generalization of traditional models by predicting 3D facial key feature points of a given input speech signal and learning robust mapping using a large-scale dataset.

[0045] This embodiment uses a NeRF-based renderer to render the head and torso separately. The head can be used as prior knowledge to guide the rendering of the torso, making the movement of the head and torso more coordinated, further improving the realism of the virtual digital human.

[0046] S4. Obtain user feedback data, and continuously optimize and adjust the lip movement prediction model based on the user feedback data to improve synchronization accuracy and naturalness.

[0047] In some embodiments, the step of obtaining user feedback data includes: Introduce a user feedback mechanism to collect user evaluation data on the synchronization effect of virtual digital population.

[0048] This embodiment provides a method for synchronizing the type of a virtual digital person, which includes the following steps: analyzing the input voice signal to obtain voice features; using the voice features as the input of a lip movement prediction model to predict corresponding lip movement parameters; driving the face model of a virtual digital person to render in real time according to the lip movement parameters, and realizing fine lip movement control; obtaining user feedback data, and continuously optimizing and adjusting the lip movement prediction model based on the user feedback data. This embodiment uses advanced technologies such as deep learning, voice analysis, and lip movement prediction to achieve accurate synchronization of the type of a virtual digital person and voice content, thereby improving the expressiveness of virtual characters and user interaction experience.

[0049] Example 2 A virtual digital population synchronization system applies the above method. For a detailed description of the method, please refer to the corresponding description in the above method embodiment, which will not be repeated here. Figure 7 As shown, the system includes a speech analysis module 51, a lip movement prediction module 52, a lip movement rendering module 53, and a feedback and optimization module 54; wherein, The speech analysis module is used to analyze the input speech signal to obtain speech features; The lip movement prediction module is used to use the speech features as input of a lip movement prediction model to predict corresponding lip movement parameters; The lip movement rendering module is used to drive the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters, and to achieve fine movement control of the lips; The feedback and optimization module is used to obtain user feedback data and continuously optimize and adjust the lip movement prediction model based on the user feedback data.

[0050] Based on the technical solution of the above embodiment, optionally, the step of analyzing the input voice signal to obtain voice features includes: Receiving an input voice signal and preprocessing the voice signal; The speech signal is converted into a text sequence using speech recognition technology, and speech features are extracted.

[0051] Based on the technical solution of the above embodiment, optionally, the lip movement prediction model is constructed based on a deep learning framework.

[0052] Based on the technical solution of the above embodiment, optionally, the training of the lip movement prediction model includes: Use the labeled speech-lip movement data to train the model and learn the mapping relationship between speech features and lip movement parameters.

[0053] Based on the technical solution of the above embodiment, optionally, the step of realizing fine lip movement control includes: Use skeletal animation or shape key technology to achieve fine movement control of lips.

[0054] Based on the technical solution of the above embodiment, optionally, the method further includes the following steps: Realize lip shape changes under various expressions and emotions.

[0055] Based on the technical solution of the above embodiment, optionally, the step of obtaining user feedback data includes: Introduce a user feedback mechanism to collect user evaluation data on the synchronization effect of virtual digital population.

[0056] Based on the technical solution of the above embodiment, optionally, the step of driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters further includes: Preprocess the input video; The preprocessed video and the speech signal are input into a lip synchronization network to generate a lip synchronization video.

[0057] Based on the technical solution of the above embodiment, optionally, the step of preprocessing the input video includes: Cut out the person from the video frame, use the semantic-guided reproduction network to modify each frame of the person's expression in the video frame according to the expression template, and generate a video with standardized expressions; Mask the mouth part in the original image; The video with the standard expression is aligned and connected with the mask processing result.

[0058] Based on the technical solution of the above embodiment, optionally, the step of inputting the preprocessed video and the speech signal into a lip synchronization network to generate a lip synchronization video includes: The alignment and connection results and the speech signal are input into the lip synchronization network, and the best mouth shape position in the audio signal and the video frame is matched through a deep learning model to achieve lip synchronization.

[0059] Based on the technical solution of the above embodiment, optionally, the step of driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters further includes: The lip-synced video is post-processed through an identity-aware face augmentation network to improve the photorealism of the synthesized face.

[0060] Based on the technical solution of the above embodiment, optionally, the step of driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters further includes: The lip-sync video is processed by super-resolution reconstruction technology to improve the video quality.

[0061] Based on the technical solution of the above embodiment, optionally, the step of driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters further includes: Reduce processing time by optimizing algorithms to ensure real-time lip sync; Post-processing sharpening technology is used to eliminate data domain offset and improve the high fidelity of the output video.

[0062] This embodiment provides a virtual digital population type synchronization system, which includes a speech analysis module, a lip movement prediction module, a lip movement rendering module, and a feedback and optimization module. This embodiment uses advanced technologies such as deep learning, speech analysis and lip movement prediction to achieve accurate synchronization of virtual digital population type and speech content, thereby improving the expression ability of virtual characters and user interaction experience.

[0063] Example 3 A computer device 600, such as Figure 8 As shown, it includes a memory 610, a processor 620, and a computer program 630 stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a virtual digital population synchronization method are implemented. For a detailed description of the method, reference may be made to the corresponding description in the above method embodiment, which will not be repeated here.

[0064] Example 4 A computer readable storage medium such as Fig. 9 As shown, a computer program is stored thereon, and when the computer program is executed by the processor, the steps of a virtual digital population type synchronization method are implemented. For a detailed description of the method, reference may be made to the corresponding description in the above method embodiment, and no further description is given here.

[0065] Example 5 A computer program product includes a computer program, and when the computer program is executed by a processor, the steps of a virtual digital population synchronization method are implemented. For a detailed description of the method, reference may be made to the corresponding description in the above method embodiment, and no further description is given here.

[0066] The number of devices and processing scales described here are used to simplify the description of the present invention. Applications, modifications and variations of the present invention will be obvious to those skilled in the art.

[0067] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

[0068] The apparatus, computer device, non-volatile computer storage medium and method provided in the embodiments of this specification correspond to each other, and therefore, the apparatus, computer device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, computer device and non-volatile computer storage medium will not be repeated here.

[0069] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software units for implementing the method and structures within the hardware component.

[0070] The systems, devices or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions in various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or more software and / or hardware.

[0071] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0072] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0073] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0074] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0075] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0076] The specification may be described in the general context of computer-executable instructions executed by a computer, such as program units. Generally, program units include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program units may be located in local and remote computer storage media, including storage devices.

[0077] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0078] The above description is only an embodiment of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of one or more embodiments of this specification.

Claims

1. A virtual digital population synchronization method, characterized in that: The following steps are involved: Analyze the input speech signal to obtain speech features; Using the speech features as input of a lip movement prediction model to predict corresponding lip movement parameters; Driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters, and realizing fine lip movement control; Obtain user feedback data, and continuously optimize and adjust the lip movement prediction model based on the user feedback data.

2. A virtual digital population synchronization method as claimed in claim 1, characterized in that: The step of analyzing the input voice signal to obtain voice features includes: Receiving an input voice signal and preprocessing the voice signal; The speech signal is converted into a text sequence using speech recognition technology, and speech features are extracted.

3. A virtual digital population synchronization method as claimed in claim 1, characterized in that: The lip movement prediction model is built based on a deep learning framework.

4. A virtual digital population synchronization method as claimed in claim 3, characterized in that: The training of the lip movement prediction model includes: Use the labeled speech-lip movement data to train the model and learn the mapping relationship between speech features and lip movement parameters.

5. A virtual digital population synchronization method as claimed in claim 1, characterized in that: The steps of achieving fine lip motion control include: Use skeletal animation or shape key technology to achieve fine movement control of lips.

6. A virtual digital population synchronization method as claimed in claim 1, characterized in that: Also includes the steps: Realize lip shape changes under various expressions and emotions.

7. A virtual digital population synchronization method as claimed in claim 1, characterized in that: The step of obtaining user feedback data comprises: Introduce a user feedback mechanism to collect user evaluation data on the synchronization effect of virtual digital population.

8. A virtual digital population synchronization method as claimed in claim 1, characterized in that: The step of driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters also includes: Preprocess the input video; The preprocessed video and the speech signal are input into a lip synchronization network to generate a lip synchronization video.

9. A virtual digital population synchronization method as claimed in claim 8, characterized in that: The step of preprocessing the input video comprises: Cut out the person from the video frame, use the semantic-guided reproduction network to modify each frame of the person's expression in the video frame according to the expression template, and generate a video with standardized expressions; Mask the mouth part in the original image; The video with the standard expression is aligned and connected with the mask processing result.

10. A virtual digital population synchronization method as claimed in claim 9, characterized in that: The step of inputting the preprocessed video and the speech signal into a lip synchronization network to generate a lip synchronization video comprises: The alignment and connection results and the speech signal are input into the lip synchronization network, and the best mouth shape position in the audio signal and the video frame is matched through a deep learning model to achieve lip synchronization.

11. A virtual digital population synchronization method as claimed in claim 8, characterized in that: The step of driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters also includes: The lip-synced video is post-processed through an identity-aware face augmentation network to improve the photorealism of the synthesized face.

12. A virtual digital population synchronization method as claimed in claim 8, characterized in that: The step of driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters also includes: The lip-sync video is processed by super-resolution reconstruction technology to improve the video quality.

13. A virtual digital population synchronization method as claimed in claim 8, characterized in that: The step of driving the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters also includes: Reduce processing time by optimizing algorithms to ensure real-time lip sync; Post-processing sharpening technology is used to eliminate data domain offset and improve the high fidelity of the output video.

14. A virtual digital population synchronization system, using the method as claimed in any one of claims 1 to 13, characterized in that: It includes speech analysis module, lip movement prediction module, lip movement rendering module, feedback and optimization module; among them, The speech analysis module is used to analyze the input speech signal to obtain speech features; The lip movement prediction module is used to use the speech features as input of a lip movement prediction model to predict corresponding lip movement parameters; The lip movement rendering module is used to drive the facial model of the virtual digital human to perform real-time rendering according to the lip movement parameters, and to achieve fine movement control of the lips; The feedback and optimization module is used to obtain user feedback data and continuously optimize and adjust the lip movement prediction model based on the user feedback data.

15. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 13 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

17. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images