Training method, device and equipment for digital human model

By using the dental keyframe model and multiple loss functions optimization training, the challenges of the digital human model in the consistency of lip and teeth characteristics are solved, and natural and realistic lip and teeth movement generation is achieved, improving the fidelity and user experience of digital human videos.

CN119031210APending Publication Date: 2024-11-26XIAMEN CHANJING TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411063574.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing video digital life technology is difficult to succeed in maintaining high consistency and unity between the lip and teeth characteristics and the original character, especially when the lip and teeth status is large, resulting in inconsistent tooth characteristics, affecting the realistic nature of the digital human model. degree and naturalness of lip and teeth movement.

Method used

By obtaining portrait video files, the preset tooth keyframe model is used to determine the keyframe as the tooth reference image, and the tooth reference image, training image and audio data are input to the video encoding network and the audio encoding network for feature extraction, and then input to decode after stitching. The network conducts model training and uses various loss function optimization models such as lip synchronous discriminator loss, L1 loss and image quality discrimination loss.

Benefits of technology

It ensures a high consistency between the lip shape and audio content when speaking, improves the fidelity and expressiveness of the model, generates natural, coherent and realistic lip movements, and enhances the authenticity of the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119031210A_ABST
    Figure CN119031210A_ABST
Patent Text Reader

Abstract

The invention discloses a digital human model training method, which comprises the following steps: acquiring a portrait video file, and determining a key frame of the portrait video file as a tooth reference image by using a preset tooth key frame model; extracting and processing each image frame in the portrait video file to obtain a training image, and extracting and processing the audio of the portrait video file to obtain audio data; respectively inputting the tooth reference image, the training image and the audio data after adjustment into a first video coding network, a second video coding network and an audio coding network for feature extraction to obtain a first feature vector, a second feature vector and an audio feature vector; and the first feature vector, the second feature vector and the audio feature vector are spliced and then input into a decoding network for model training, and a digital human model is obtained. The fidelity of the model can be improved so as to generate a natural, coherent and vivid digital human video of lip and tooth actions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a training method, device and equipment for a digital human model. Background Art

[0002] In the existing video digital human generation technology, although it has become a common working paradigm to process audio information and reference portrait information separately, use audio coding network and video coding network for feature encoding, and then splice the two into the decoding network to generate speech lip shape aligned with audio, this method still faces significant challenges in maintaining the high consistency and unity of lip and tooth features with the original person. However, the existing technology mainly reconstructs the mouth shape features by establishing a loss function of the lip and tooth part to try to ensure that the generated lip and tooth image is as consistent as possible with the lip and tooth of the current frame, but this method has a fundamental flaw: it cannot effectively handle the significant difference in the lip and tooth state between the current reference frame and the frame that needs to be driven by audio. In particular, when the lip and tooth state (such as closed mouth) in the reference frame is significantly different from the lip and tooth state required to be driven by audio, the model often generates lip and tooth images based on its own experience due to the lack of effective reference information, resulting in inconsistencies between the generated teeth and the teeth of the real person in terms of size, shape, number of particles, etc. This inconsistency is particularly obvious in continuous image frames, because the model may independently generate tooth images based on the audio information of each frame, rather than generating them coherently based on the inherent characteristics of the original person's teeth. This not only affects the realism of the digital human model, but also destroys the naturalness and unity of lip and tooth movements. Summary of the invention

[0003] In view of this, the purpose of the present invention is to propose a training method, device and equipment for a digital human model, aiming to solve the problem of how to ensure that the lip and tooth features generated in continuous image frames always maintain a high degree of consistency and unity with the original person, so as to generate a digital human video with natural, coherent and realistic lip and tooth movements.

[0004] To achieve the above object, the present invention provides a method for training a digital human model, the method comprising:

[0005] Acquire a portrait video file, and use a preset tooth key frame model to determine a key frame of the portrait video file as a tooth reference image;

[0006] Extracting and processing each image frame in the portrait video file to obtain a training image, and extracting and processing the audio of the portrait video file to obtain audio data;

[0007] Inputting the adjusted tooth reference image, the training image, and the audio data into a first video coding network, a second video coding network, and an audio coding network for feature extraction, respectively, to obtain a first feature vector, a second feature vector, and an audio feature vector;

[0008] The first feature vector, the second feature vector and the audio feature vector are concatenated and input into a decoding network for model training to obtain a digital human model for generating a digital human video with aligned portrait and audio.

[0009] Preferably, the step of using a preset tooth key frame model to determine the key frame of the portrait video file as a tooth reference image includes:

[0010] Read the image frames in the portrait video file frame by frame and input them into the tooth key frame model for scoring to obtain a score corresponding to each image frame, wherein the tooth key frame model determines the category to which each image frame belongs according to the divided categories, and each category corresponds to a score;

[0011] All frames are sorted from high to low according to the scores, and the first N image frames with the highest scores are taken as key frames to obtain the tooth reference image.

[0012] Preferably, the training process of the tooth key frame model includes:

[0013] By acquiring a tooth image dataset, and classifying and labeling the tooth image dataset according to the degree of tooth exposure, multi-classification training data is obtained;

[0014] The multi-classification training data is input into the model architecture based on resnext101_32x32d for training to obtain the tooth key frame model.

[0015] Preferably, extracting and processing each image frame in the portrait video file to obtain a training image includes:

[0016] Using a face detection network to identify the face frame and face key point information of each image frame, calculating the first distance from the center of the eyebrows to the chin according to the face frame and the face key point information, and cropping after expanding the first distance outward from the center point of the mouth by a preset multiple to obtain a plurality of face information images, and adjusting each of the face information images to a preset size to obtain a face input image;

[0017] An image frame of the same face and a frame to be generated are randomly extracted from the face input image, and a mask operation of the mouth region is performed on the frame to be generated to obtain the training image.

[0018] Preferably, extracting and processing the audio of the portrait video file to obtain audio data includes:

[0019] The audio is encoded and represented using Mel-atrogram to obtain the audio data in which 5 consecutive frames of audio information are represented by 80*16 atrium values ​​each.

[0020] Preferably, the adjusted teeth reference image is obtained by segmenting the lip region in the teeth reference image through a segmentation model.

[0021] Preferably, the training process of the digital human model includes optimizing the model using lip synchronization discriminator loss, L1 loss, image quality discrimination loss and vgg perception loss.

[0022] To achieve the above object, the present invention further provides a training device for a digital human model, the device comprising:

[0023] A key frame determination unit, used for acquiring a portrait video file, and determining a key frame of the portrait video file as a tooth reference image using a preset tooth key frame model;

[0024] A data processing unit, used to extract and process each image frame in the portrait video file to obtain a training image, and to extract and process the audio of the portrait video file to obtain audio data;

[0025] A feature extraction unit, used for inputting the adjusted tooth reference image, the training image, and the audio data into a first video coding network, a second video coding network, and an audio coding network for feature extraction, to obtain a first feature vector, a second feature vector, and an audio feature vector;

[0026] The model training unit is used to splice the first feature vector, the second feature vector and the audio feature vector and input them into a decoding network for model training to obtain a digital human model for generating a digital human video with aligned portrait and audio.

[0027] In order to achieve the above objectives, the present invention also proposes a training device for a digital human model, comprising a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a training method for a digital human model as described in the above embodiment.

[0028] In order to achieve the above objectives, the present invention further proposes a computer-readable storage medium, on which a computer program is stored. The computer program is executed by a processor to implement the steps of a digital human model training method as described in the above embodiment.

[0029] In order to achieve the above-mentioned object, the present invention further proposes a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of a digital human model training method as described in the above-mentioned embodiment.

[0030] Beneficial effects:

[0031] The above scheme, by using the preset tooth keyframe model to determine the keyframes in the portrait video as the tooth reference image, can accurately capture the dynamic changes of teeth and more effectively utilize the inherent characteristics of the original person's teeth when processing audio information and reference portrait information, which is crucial for generating natural and smooth lip movements. It ensures that the lip shape of the digital human model is highly consistent with the audio content when speaking, thereby improving the model's fidelity and expressiveness, and generating digital human videos with natural, coherent and realistic lip and tooth movements.

[0032] The above scheme uses the tooth keyframe model to score each frame and sort the image frames with the highest scores as keyframes, which effectively reduces the processing of redundant data, improves the quality of training data, and thus improves the efficiency and accuracy of model training. At the same time, the tooth keyframe model is trained based on the resnext101_32x32d model architecture, which has strong feature extraction capabilities and can more accurately identify the degree of tooth exposure, providing strong support for subsequent feature extraction and model training.

[0033] The above solution achieves precise alignment of audio and video by inputting dental reference images, training images and audio data into different encoding networks for feature extraction, and concatenating these feature vectors and inputting them into the decoding network for model training. This allows the digital human model to fully synchronize its lip movements with the audio content when speaking, enhancing the realism of the user experience.

[0034] In the above scheme, multiple loss functions such as lip synchronization discriminator loss, L1 loss, image quality discrimination loss and vgg perception loss are used to optimize the model during the model training process. These loss functions evaluate and adjust the model from different angles, ensuring that the model can gradually optimize various performance indicators during the training process and finally generate a high-quality digital human model. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A schematic flow chart of a method for training a digital human model provided by one embodiment of the present invention.

[0037] Figure 2 A schematic diagram of the overall model network structure provided for one embodiment of the present invention.

[0038] Figure 3 A schematic diagram of a network data flow provided by an embodiment of the present invention.

[0039] Figure 4 A schematic diagram of the structure of a digital human model training device provided by one embodiment of the present invention.

[0040] The realization of the purpose, functional features and advantages of the invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0042] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0043] The present invention is described in detail below with reference to the embodiments.

[0044] Reference Figure 1 FIG. 1 is a flow chart of a method for training a digital human model provided by an embodiment of the present invention.

[0045] In this embodiment, the method includes:

[0046] S11, obtaining a portrait video file, and using a preset tooth key frame model to determine a key frame of the portrait video file as a tooth reference image.

[0047] Furthermore, in step S11, the step of using a preset tooth key frame model to determine the key frame of the portrait video file as a tooth reference image includes:

[0048] S11-1, reading the image frames in the portrait video file frame by frame and inputting them into the tooth key frame model for scoring to obtain a score corresponding to each image frame, wherein the tooth key frame model determines the category to which each image frame belongs according to the divided categories, and each category corresponds to a score;

[0049] S11-2, sorting all frames from high to low according to the scores, taking the first N image frames with the highest scores as key frames, and obtaining the tooth reference image.

[0050] Furthermore, the training process of the tooth key frame model includes:

[0051] By acquiring a tooth image dataset, and classifying and labeling the tooth image dataset according to the degree of tooth exposure, multi-classification training data is obtained;

[0052] The multi-classification training data is input into the model architecture based on resnext101_32x32d for training to obtain the tooth key frame model.

[0053] In this embodiment, each acquired portrait video file is read frame by frame to extract high-quality dental key frames (image frames with clear teeth and more exposed features), and the top k image frames of each image frame in the portrait video file in the dental key frame model are taken as dental reference images, such as k=2 in the specific implementation of data sampling.

[0054] By obtaining a dataset of teeth images, the dataset is divided into categories and annotated and scored according to the degree of tooth exposure, and multi-classification training data is obtained, such as the first category of closed mouth is 0 points, the second category of less than 3 exposed teeth is 2 points, the third category of relatively complete exposure of 3-5 is 4 points, the fourth category of relatively complete exposure of 6-8 is 6 points, the fifth category of relatively complete exposure of 8-10 is 8 points, and the sixth category of basically full exposure of the upper and lower rows of teeth is 10 points. Based on this, the dataset is divided into 6 intervals, with quality of 0 points, 2 points, 4 points, 6 points, 8 points, and 10 points, respectively, and the dataset is divided into 6 categories, 0 points is the worst quality, and 10 points is the best quality. By using the open source network structure "resnext101_32x32d" to train a 6-category model, after the training is completed, the weight of this model is the tooth key frame model. When a portrait picture is input, the model gives a category, and the category corresponds to a score. The category belonging to the 0-point interval category is low quality, and the category belonging to the 10-point interval category is high quality.

[0055] S12, extracting and processing each image frame in the portrait video file to obtain a training image, and extracting and processing the audio of the portrait video file to obtain audio data.

[0056] Furthermore, in step S12, extracting and processing each image frame in the portrait video file to obtain a training image includes:

[0057] S12-1, using a face detection network to identify the face frame and face key point information of each image frame, calculating a first distance from the center of the eyebrows to the chin based on the face frame and the face key point information, and cropping after expanding the first distance outward from the center of the mouth by a preset multiple to obtain a plurality of face information images, and adjusting each of the face information images to a preset size to obtain a face input image;

[0058] An image frame of the same face and a frame to be generated are randomly extracted from the face input image, and a mask operation of the mouth region is performed on the frame to be generated to obtain the training image.

[0059] Furthermore, in step S12, the audio of the portrait video file is extracted and processed to obtain audio data, including:

[0060] S12-2, using Mel-atrogram to encode the audio and obtain the audio data in which 5 consecutive frames of audio information are represented by 80*16 atrium values ​​each.

[0061] In this embodiment, each image frame in the portrait video file is read frame by frame, and the face detection network (face_alignment) is used to identify the face frame and 68 face key point information of the image frame. By calculating the distance (length) from the center of the eyebrows to the chin of the face, the face information image is cropped out by expanding 1.n times the length size from the center of the mouth, and then resized to a width and height size of 192*192, that is, the size image input by the model, to obtain a training image. The audio in the portrait video file is represented by audio encoding using Mel atlas (mel), where each 80*16 atlas value represents the audio information of 5 consecutive frames. In order to generate the continuity of the picture, the audio information of 5 consecutive frames is usually used to correspond to 1 frame of the picture, that is, when the current frame of the picture is generated, it can perceive the audio information of the two frames before and after.

[0062] S13, inputting the adjusted tooth reference image, the training image, and the audio data into the first video coding network, the second video coding network, and the audio coding network respectively for feature extraction to obtain a first feature vector, a second feature vector, and an audio feature vector.

[0063] Furthermore, the adjusted tooth reference image is obtained by segmenting the lip region in the tooth reference image through a segmentation model.

[0064] Reference Figure 2 In this embodiment, the digital human model includes a feature encoding layer and a decoding layer (generation network), wherein the feature encoding layer includes information encoding of three feature channels, two of which are encoding networks for image frames in a portrait video file (video encoding network 1, video encoding network 2) and one audio encoding network corresponding to the audio. By passing a 192*192*6 tooth reference image (first use a segmentation model to segment the lip area in the image, that is, the segmented lip area image is used as the adjusted tooth reference image) through a visual coding network 1 (such as a typical wav2lip network) for feature extraction, a high-dimensional first feature vector of length 512 is obtained; specifically: when the input data is a 192*192*6 picture, the encoding process of each layer is as follows: first, 192*192*6 is transposed to 6*192*192, and then through the first layer: 16*192*192, the second layer: 32*96*96, the third layer: 64*48*48, the fourth layer: 128*24*24, the fifth layer: 256*12*12, the sixth layer: 512*6*6, the seventh layer: 512*3*3, the eighth layer: 512*1*1, and finally, 512*1*1 is removed by one dimension to obtain a 512-length feature vector.

[0065] The 192*192*6 training image is subjected to feature extraction through the visual coding network 2 (such as a typical wav2lip network) to obtain a high-dimensional feature vector of length 512; the processing process can refer to the above-mentioned visual coding network 1. Among them, the training image includes a random reference frame of the character (the random reference frame is an image frame randomly extracted from the portrait video file) and a frame to be generated (the generated result obtained after model decoding is a label relationship with the frame to be generated during the training process, that is, the relationship between the generated y and the real y), and the mouth area of ​​the frame to be generated is masked. The goal of network learning is to generate a mouth shape that matches the voice and character in the mask area. The specific method of masking is to connect the coordinates of the key points of the face into a closed image and perform a certain expansion to obtain a mask area, and the two frames are superimposed to obtain a 192*192*6 image input.

[0066] The 80*16 mel audio data is passed through an audio coding network (using a classic audio coding network) for feature extraction to obtain a high-dimensional audio feature vector of length 512. The input audio here corresponds one-to-one to the audio corresponding to the input frame to be generated in the original video file. Specifically, the intermediate encoding process after the 1*80*16 audio data passes through the network includes: the first layer: 32*80*16, the second layer: 64*27*16, the third layer: 128*9*5, the fourth layer: 256*3*3, the fifth layer: 512*1*1, and finally the 1*1*512 is removed by one dimension to obtain a 512-length feature vector.

[0067] S14, concatenating the first feature vector, the second feature vector, and the audio feature vector and inputting the concatenated features into a decoding network for model training, thereby obtaining a digital human model for generating a digital human video with aligned portrait and audio.

[0068] Furthermore, the training process of the digital human model includes optimizing the model using lip synchronization discriminator loss, L1 loss, image quality discrimination loss and vgg perception loss.

[0069] Reference Figure 3 As shown. In this embodiment, when decoding to generate the lip shape corresponding to the audio, the 512-dimensional first feature vector, the second feature vector and the audio feature vector are spliced ​​to obtain 1536 feature information, and then decoded in the decoding network (generation network) to obtain a 192*192*3 character lip shape driven by the audio and consistent with the audio. The decoder uses the decoding network in the classic wav2lip network. Specifically, the process of splicing the first feature vector, the second feature vector and the audio feature vector and inputting them into the decoding network includes:

[0070] First, input the result of audio encoding: 512*1*1, and get the output result after the first layer of the decoding network: 512*1*1;

[0071] The output of the first layer is concatenated with the eighth layer results of the visual coding network 1 and the visual coding network 2 to obtain 1536*1*1, and the concatenated result is input into the second layer of the decoding network to obtain: 512*3*3;

[0072] The second layer output is concatenated with the seventh layer results of visual coding network 1 and visual coding network 2 to obtain 1536*3*3, and the concatenated result is input into the third layer of the decoding network to obtain: 512*6*6;

[0073] The output of the third layer is concatenated with the sixth layer results of the visual coding network 1 and the visual coding network 2 to obtain 1536*6*6, and the concatenated result is input into the fourth layer of the decoding network to obtain: 512*12*12;

[0074] The output of the fourth layer is concatenated with the fifth layer results of the visual coding network 1 and the visual coding network 2 to obtain 1024*12*12, and the concatenated result is input into the fifth layer of the decoding network to obtain: 256*24*24;

[0075] The fifth layer output is concatenated with the fourth layer results of visual coding network 1 and visual coding network 2 to obtain 512*24*24, and the concatenated result is input into the sixth layer of the decoding network to obtain: 256*48*48;

[0076] The sixth layer output is concatenated with the third layer results of visual coding network 1 and visual coding network 2 to obtain 384*48*48, and the concatenated result is input into the seventh layer of the decoding network to obtain: 128*96*96;

[0077] The seventh layer output is concatenated with the second layer results of visual coding network 1 and visual coding network 2 to obtain 192*96*96, and the concatenated result is input into the eighth layer of the decoding network to obtain: 64*192*192;

[0078] The output of the eighth layer is concatenated with the first layer results of the visual coding network 1 and the visual coding network 2 to obtain 96*192*192, and the concatenated result is input into the ninth layer of the decoding network to obtain: 3*192*192.

[0079] The output of the ninth layer is the decoding result, which participates in the loss calculation during training and is the output result during inference.

[0080] The loss function used in model training is a typical multi-loss function that works with wav2lip, which includes syncne_losst (lip synchronization discriminator), L1 loss, image quality discrimination loss, and vgg perception loss. In summary, the clarity of the digital human's speech driven by a fixed reference frame and the stability of teeth between consecutive frames have been greatly improved in this case.

[0081] Reference Figure 3 FIG. 1 is a schematic diagram of the structure of a training device for a digital human model provided by an embodiment of the present invention.

[0082] In this embodiment, the device 20 includes:

[0083] A key frame determination unit 21 is used to obtain a portrait video file, and determine a key frame of the portrait video file as a tooth reference image using a preset tooth key frame model;

[0084] A data processing unit 22 is used to extract and process each image frame in the portrait video file to obtain a training image, and to extract and process the audio of the portrait video file to obtain audio data;

[0085] A feature extraction unit 23 is used to input the adjusted tooth reference image, the training image, and the audio data into a first video coding network, a second video coding network, and an audio coding network for feature extraction to obtain a first feature vector, a second feature vector, and an audio feature vector;

[0086] The model training unit 24 is used to splice the first feature vector, the second feature vector and the audio feature vector and input them into a decoding network for model training to obtain a digital human model for generating a digital human video with aligned portrait and audio.

[0087] Each unit module of the device 20 can respectively execute the corresponding steps in the above method embodiment, so each unit module will not be described in detail here. Please refer to the description of the above corresponding steps for details.

[0088] The embodiment of the present invention also provides a digital human model training device, the device includes the digital human model training device as described above, wherein the digital human model training device can be used Figure 3 The structure of the embodiment can be executed accordingly. Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0089] The device includes: a mobile phone, a digital camera, a tablet computer, or other devices with a camera function, or a device with an image processing function, or a device with an image display function. The device may include components such as a memory, a processor, an input unit, a display unit, and a power supply.

[0090] Among them, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory may also include a memory controller to provide the processor and the input unit with access to the memory.

[0091] The input unit can be used to receive input digital or character or image information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit of this embodiment includes not only a camera, but also a touch-sensitive surface (such as a touch display) and other input devices.

[0092] The display unit can be used to display information input by the user or information provided to the user and various graphical user interfaces of the device, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit may include a display panel, and optionally, the display panel may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface may cover the display panel, and when the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor to determine the type of touch event, and then the processor provides a corresponding visual output on the display panel according to the type of touch event.

[0093] The embodiment of the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory in the above embodiment; or a computer-readable storage medium that exists independently and is not installed in a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement Figure 1 The computer readable storage medium may be a read-only memory, a disk or an optical disk, etc.

[0094] The embodiment of the present invention also provides a computer program product, including a computer program / instruction, which is loaded and executed by a processor to implement Figure 1 A training method for a digital human model is shown.

[0095] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, equipment embodiment and storage medium embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0096] Furthermore, in this document, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0097] The above description shows and describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention, through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not depart from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A method for training a digital human model, characterized in that: The method comprises: Acquire a portrait video file, and use a preset tooth key frame model to determine a key frame of the portrait video file as a tooth reference image; Extracting and processing each image frame in the portrait video file to obtain a training image, and extracting and processing the audio of the portrait video file to obtain audio data; Inputting the adjusted tooth reference image, the training image, and the audio data into a first video coding network, a second video coding network, and an audio coding network for feature extraction, respectively, to obtain a first feature vector, a second feature vector, and an audio feature vector; The first feature vector, the second feature vector and the audio feature vector are concatenated and input into a decoding network for model training to obtain a digital human model for generating a digital human video with aligned portrait and audio.

2. A digital human model training method according to claim 1, characterized in that: The method of using a preset tooth key frame model to determine the key frame of the portrait video file as a tooth reference image includes: Read the image frames in the portrait video file frame by frame and input them into the tooth key frame model for scoring to obtain a score corresponding to each image frame, wherein the tooth key frame model determines the category to which each image frame belongs according to the divided categories, and each category corresponds to a score; All frames are sorted from high to low according to the scores, and the first N image frames with the highest scores are taken as key frames to obtain the tooth reference image.

3. A digital human model training method according to claim 1, characterized in that: The training process of the tooth key frame model includes: By acquiring a tooth image dataset, and classifying and labeling the tooth image dataset according to the degree of tooth exposure, multi-classification training data is obtained; The multi-classification training data is input into the model architecture based on resnext101_32x32d for training to obtain the tooth key frame model.

4. A digital human model training method according to claim 1, characterized in that: The extracting and processing each image frame in the portrait video file to obtain a training image includes: Using a face detection network to identify the face frame and face key point information of each image frame, calculating the first distance from the center of the eyebrows to the chin according to the face frame and the face key point information, and cropping after expanding the first distance outward from the center point of the mouth by a preset multiple to obtain a plurality of face information images, and adjusting each of the face information images to a preset size to obtain a face input image; An image frame of the same face and a frame to be generated are randomly extracted from the face input image, and a mask operation of the mouth region is performed on the frame to be generated to obtain the training image.

5. The method for training a digital human model according to claim 1, characterized in that: The step of extracting and processing the audio of the portrait video file to obtain audio data includes: The audio is encoded and represented using Mel-atrogram to obtain the audio data in which 5 consecutive frames of audio information are represented by 80*16 atrium values ​​each.

6. A digital human model training method according to claim 1, characterized in that: The adjusted tooth reference image is obtained by segmenting the lip region in the tooth reference image using a segmentation model.

7. A digital human model training method according to claim 1, characterized in that: The training process of the digital human model includes optimizing the model using lip synchronization discriminator loss, L1 loss, image quality discrimination loss and vgg perception loss.

8. A training device for a digital human model, characterized in that: The device comprises: A key frame determination unit, used for acquiring a portrait video file, and determining a key frame of the portrait video file as a tooth reference image using a preset tooth key frame model; A data processing unit, used to extract and process each image frame in the portrait video file to obtain a training image, and to extract and process the audio of the portrait video file to obtain audio data; A feature extraction unit, used for inputting the adjusted tooth reference image, the training image, and the audio data into a first video coding network, a second video coding network, and an audio coding network for feature extraction, to obtain a first feature vector, a second feature vector, and an audio feature vector; The model training unit is used to splice the first feature vector, the second feature vector and the audio feature vector and input them into a decoding network for model training to obtain a digital human model for generating a digital human video with aligned portrait and audio.

9. A training device for a digital human model, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a digital human model training method as claimed in any one of claims 1 to 7.

10. A computer program product, comprising a computer program / instruction, wherein the computer program / instruction is executed by the processor to implement the steps of a digital human model training method as described in any one of claims 1 to 7.

Citation Information

Cited By

  • Video generation method and device, equipment and medium

    CN120186433A

  • Video generation method, device, equipment and medium

    CN120186433B