Digital human batch generation method and device, equipment, medium and program product
By acquiring the digital human feature parameter information set to generate an initial video, and performing facial optimization and motion matching processing, the problem of low efficiency in batch generation of digital humans is solved, and efficient and automated digital human video generation is achieved.
Patent Information
- Application Number
- CN202511604154.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-04
- Publication Date
- 2026-02-10
AI Technical Summary
Existing technologies for generating digital humans in batches are characterized by low efficiency, long cycles, and high costs, making it difficult to scale up production and resulting in low efficiency in mass production.
By acquiring a set of digital human feature parameters, an initial digital human video is generated, and the facial feature content is optimized. A pre-trained facial parameter mapping model is used to repair facial details, and action segments are connected to generate a high-quality digital human video.
It enables efficient and automated batch generation of digital human videos, improves production efficiency, avoids unnatural changes in digital human movements, and achieves mass production.
Smart Images
Figure CN121509769A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] Embodiments of the present disclosure relate to the technical field of digital humans, and in particular, to a digital human batch generation method, device, equipment, medium and program product. BACKGROUND
[0002] At present, with the continuous development of digital humans, digital humans are increasingly widely applied in various scenarios. How to batch generate digital humans has become one of the current main development directions. For batch generation of digital humans, the commonly used way is to generate digital humans for various digital human requirements in the way of manual shooting and manual subsequent processing.
[0003] However, the inventors have found that when the above method is used, the following technical problems often exist: Manual digital human production has the problems of low efficiency, long cycle, high cost and difficulty in scaling, resulting in low efficiency of batch production. SUMMARY
[0004] The summary part of the present disclosure is used to introduce the concepts in a brief form, which will be described in detail in the specific embodiments part. The summary part of the present disclosure is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.
[0005] Some embodiments of the present disclosure provide a digital human batch generation method, device, equipment, medium and program product to solve the technical problems mentioned in the background part.
[0006] In a first aspect, some embodiments of the present disclosure provide a digital human batch generation method, comprising: obtaining a set of digital human feature parameter information selected from a digital human feature parameter library for digital human batch generation; for each digital human feature parameter information, performing the following generation steps: generating an initial digital human video corresponding to the digital human feature parameter information; optimizing the face feature content corresponding to the initial digital human video to obtain a digital human optimized video; and performing action connection processing between each digital human action segment corresponding to the digital human optimized video to obtain a digital human video.
[0007] Optionally, the face feature content corresponding to the initial digital human video is optimized to obtain a digital human optimized video, including: extracting audio feature representation information corresponding to the initial digital human video; inputting the audio feature representation information into a pre-trained first face parameter mapping model to obtain digital human face parameter information; performing face detail repair on an initial digital human face image rendered from the digital human face parameter information to obtain a digital human face image; and optimizing the face feature content corresponding to the initial digital human video according to the digital human face image to obtain the digital human optimized video.
[0008] Optionally, the first and last frames of each digital human action segment are the same, and the motion connection processing between each digital human action segment corresponding to the digital human optimized video is performed to obtain a digital human video, including: selecting a first digital human action segment corresponding to a listening state type from each digital human action segment to obtain a first digital human action segment set; for each first digital human action segment, the following replacement steps are performed: determining an action transition point corresponding to the first digital human action segment in response to determining that the next action segment corresponding to the first digital human action segment is a segment of a non-listening state type; generating a sequence of digital human frames from the action transition point to the next action segment using a video frame interpolation algorithm; replacing the segment after the action transition point in the first digital human action segment with the sequence of digital human frames to obtain an action replacement segment; and generating the digital human video according to each action replacement segment obtained.
[0009] Optionally, the determination of the action transition point corresponding to the first digital human action segment includes: for each digital human frame sub-sequence in the first digital human action segment, determining similarity information between each digital human frame in the digital human frame sub-sequence and the first and last frames to obtain a similarity information set; determining a digital human frame sub-sequence corresponding to the similarity information set that satisfies a target similarity condition from the first digital human action segment as a target digital human frame sub-sequence; and determining the action transition point according to the target digital human frame sub-sequence.
[0010] Optionally, the method further includes: for each digital human video in the obtained digital human video set, performing an adjustment step: performing quality auditing on the digital human video according to digital human feature parameter information corresponding to the digital human video to obtain an auditing result; in response to the auditing result indicating that the digital human video has a quality problem, performing video adjustment on the digital human video according to the auditing result and the digital human feature parameter information to obtain an adjusted digital human video; and performing video publishing on the adjusted digital human video.
[0011] Optionally, the first face parameter mapping model is trained by the following steps: obtaining a target training data set, wherein the target training data includes: digital human audio and digital human video; for each target training data, performing the following generation steps: determining high-quality speech representation information corresponding to the digital human audio included in the target training data; extracting face parameter information corresponding to the digital human video included in the target training data; training a second initial face parameter mapping model according to the obtained set of high-quality speech representation information and the set of face parameter information, to obtain a second face parameter mapping model; taking the second face parameter mapping model as a teacher model, and taking the first initial face parameter mapping model as a student model to be trained, and training the first initial face parameter mapping model to obtain the first face parameter mapping model.
[0012] In a second aspect, some embodiments of the present disclosure provide a digital human batch generation apparatus, comprising: an obtaining unit configured to obtain a set of digital human feature parameter information selected from a digital human feature parameter library for digital human batch generation; and an execution unit configured to, for each digital human feature parameter information, perform the following generation steps: generating an initial digital human video corresponding to the digital human feature parameter information; optimizing face feature content corresponding to the initial digital human video to obtain a digital human optimized video; and performing action connection processing between each digital human action segment corresponding to the digital human optimized video to obtain a digital human video.
[0013] Optionally, the execution unit can be configured to: extract audio feature representation information corresponding to the initial digital human video; input the audio feature representation information into a pre-trained first face parameter mapping model to obtain digital human face parameter information; perform face detail repair on an initial digital human face image rendered from the digital human face parameter information to obtain a digital human face image; and optimize the face feature content corresponding to the initial digital human video according to the digital human face image to obtain the digital human optimized video.
[0014] Optionally, each digital human action segment corresponds to the same first and last frames; and the execution unit can be configured to: filter out a first digital human action segment corresponding to a listening state type from each digital human action segment to obtain a set of first digital human action segments; for each first digital human action segment, perform the following replacement steps: determining an action transition point corresponding to the first digital human action segment in response to determining that a next action segment corresponding to the first digital human action segment is a segment of a non-listening state type; generating a sequence of digital human frames from the action transition point to the next action segment using a video frame interpolation algorithm; replacing a segment after the action transition point in the first digital human action segment with the sequence of digital human frames to obtain an action replacement segment; and generating the digital human video according to each action replacement segment obtained.
[0015] Optionally, the execution unit can be configured to: for each sub-sequence of digital human frames in the first sub-sequence of digital human action, determine the similarity information between each digital human frame in the sub-sequence of digital human frames and the first and last frames, to obtain a set of similarity information; determine a sub-sequence of digital human frames that meets a target similarity condition from the set of similarity information as a target sub-sequence of digital human frames; and determine the action transition point according to the target sub-sequence of digital human frames.
[0016] Optionally, the device further comprises: for each digital human video in the obtained set of digital human videos, performing an adjustment step: performing quality auditing on the digital human video according to the digital human feature parameter information corresponding to the digital human video to obtain an auditing result; in response to the auditing result indicating that the digital human video has a quality problem, performing video adjustment on the digital human video according to the auditing result and the digital human feature parameter information to obtain an adjusted digital human video; and performing video publishing on the adjusted digital human video.
[0017] In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect.
[0018] In a fourth aspect, some embodiments of the present disclosure provide a computer readable medium having a computer program stored thereon, wherein the program is executed by a processor to implement the method described in any implementation manner of the first aspect.
[0019] In a fifth aspect, some embodiments of the present disclosure provide a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the method described in any implementation manner of the first aspect.
[0020] The above various embodiments of the present disclosure have the following beneficial effects: through the digital human batch generation method of some embodiments of the present disclosure, a plurality of digital human videos required can be efficiently and automatically batch generated. Specifically, the reason why the related digital human video is not batch generated is not efficient is that manual digital human production has problems such as low efficiency, long cycle, high cost, and difficulty in scaling, resulting in low efficiency of batch production. Based on this, the digital human batch generation method of some embodiments of the present disclosure first acquires a set of digital human feature parameter information selected from a digital human feature parameter library for digital human batch generation, to obtain parameter content for subsequent digital human batch production. Then, for each digital human feature parameter information, the following generation steps are performed: first, an initial digital human video corresponding to the above digital human feature parameter information can be preliminarily and accurately generated. Second, the face feature content corresponding to the above initial digital human video is optimized to obtain a digital human optimized video, which can realize fine processing of face detail content and obtain a more accurate digital human video. Third, the action connection between each digital human action segment corresponding to the above digital human optimized video is processed to obtain a digital human video, which can effectively avoid unnatural problems of the digital human when changing actions. In summary, through the selection of digital human feature parameter information, the generation of initial digital human video, and multi-faceted optimization, the digital production process is automatically processed, which can realize the batch generation of digital human videos on the basis of ensuring production efficiency. BRIEF DESCRIPTION OF DRAWINGS
[0021] The above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description when taken in conjunction with the accompanying drawings, in which like reference characters indicate like elements throughout the drawings. It is to be understood that the drawings are schematic, and elements and features are not necessarily drawn to scale.
[0022] Figure 1 is a schematic diagram of an application scenario of the digital human batch generation method according to some embodiments of the present disclosure; Figure 2 is a flowchart of some embodiments of the digital human batch generation method according to the present disclosure; Figure 3 is a flowchart of some other embodiments of the digital human batch generation method according to the present disclosure; Figure 4 is a structural schematic diagram of some embodiments of the digital human batch generation apparatus according to the present disclosure; Figure 5 is a structural schematic diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION
[0023] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be interpreted as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be more thoroughly and completely understood. It should be understood that the drawings of the present disclosure are only for illustrative purposes and are not intended to limit the scope of protection of the present disclosure.
[0024] In addition, it should be further noted that only parts related to the present application are shown in the drawings for ease of description. The embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.
[0025] It should be noted that the terms “first”, “second”, and the like mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.
[0026] It should be noted that the terms “one”, “multiple” mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless otherwise explicitly stated in the context, it should be understood as “one or more”.
[0027] The names of the messages or information exchanged between the devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.
[0028] The collection, storage and use of user personal information (such as digital human feature parameter information) involved in the present disclosure should be performed by the relevant organization or individual to fulfill the obligations including conducting personal information security impact assessment, fulfilling the notification obligations to the personal information subject, obtaining the prior authorization consent of the personal information subject, etc.
[0029] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.
[0030] Figure 1 is a schematic diagram of one application scenario of a digital human batch generation method according to some embodiments of the present disclosure.
[0031] In Figure 1In the application scenario of the digital human video generation method, first, the electronic device 101 can obtain the digital human characteristic parameter information set 103 selected from the digital human characteristic parameter library 102 for batch generation of digital humans. In this application scenario, the digital human characteristic parameter information set 103 can include: “face feature 1, clothing feature 1, style feature 1”, “face feature 2, personality feature 1, style feature 2”, and “face feature 3, personality feature 2, style feature 3”. Then, for each digital human characteristic parameter information, the electronic device 101 can perform the following generation steps: first, generate the initial digital human video corresponding to the above-mentioned digital human characteristic parameter information. In this application scenario, for the digital human characteristic parameter information “face feature 1, clothing feature 1, style feature 1”, the corresponding initial digital human video is initial digital human video 1. For the digital human characteristic parameter information “face feature 2, personality feature 1, style feature 2”, the corresponding initial digital human video is initial digital human video 2. For the digital human characteristic parameter information “face feature 3, personality feature 2, style feature 3”, the corresponding initial digital human video is initial digital human video 3. Second, optimize the face feature content corresponding to the above-mentioned initial digital human video to obtain a digital human optimization video. In this application scenario, for the initial digital human video 1, the corresponding digital human optimization video is digital human optimization video 1. For the initial digital human video 2, the corresponding digital human optimization video is digital human optimization video 2. For the initial digital human video 3, the corresponding digital human optimization video is digital human optimization video 3. Third, perform motion connection processing between each digital human motion segment corresponding to the above-mentioned digital human optimization video to obtain a digital human video. In this application scenario, for the digital human optimization video 1, the corresponding digital human video is digital human video 1. For the digital human optimization video 2, the corresponding digital human video is digital human video 2. For the digital human optimization video 3, the corresponding digital human video is digital human video 3.
[0032] It should be noted that the electronic device 101 described above can be hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster composed of multiple servers or terminal devices, or as a single server or a single terminal device. When the electronic device is software, it can be installed in the hardware devices listed above. It can be implemented as, for example, multiple software or software modules for providing distributed services, or as a single software or software module. No specific limitation is made here.
[0033] It should be understood that Figure 1 The number of electronic devices in the above-mentioned application scenarios is only illustrative. Depending on the implementation needs, there can be any number of electronic devices.
[0034] Continuing to refer to the above-mentioned application scenarios Figure 2FIG. 200 shows a flow 200 of some embodiments of the digital human batch generation method according to the present disclosure. The digital human batch generation method comprises the following steps: Step 201, obtaining a set of digital human characteristic parameter information selected from a digital human characteristic parameter library for digital human batch generation.
[0035] In some embodiments, the execution subject of the above-mentioned digital human batch generation method (for example Figure 1 The electronic device 101 shown) can obtain a set of digital human characteristic parameter information selected from a digital human characteristic parameter library for digital human batch generation through wired connection or wireless connection. The digital human characteristic parameter library is a storage library that stores parameter information related to various styles of digital human characteristics. For example, for the face of a digital human, different styles of face parameter information (e.g., parameter information of different face contours) can be stored. For example, for the body shape characteristics of a digital human, different styles of body shape parameter information can be stored. In practice, the types of characteristics that exist in the digital human characteristic parameter library can include but are not limited to at least one of the following: facial features, body shape characteristics, clothing style characteristics, expression characteristics, and action characteristics. The digital human characteristic parameter information in the set of digital human characteristic parameter information can be the selected parameter information for generating a digital human for the digital human characteristic parameter information. In practice, the digital human characteristic parameter information can include the style content of the corresponding style under each characteristic type. For example, the digital human characteristic parameter information can be a digital human with a round face, a slightly overweight body, and wearing a yellow coat and black shorts. In practice, each digital human characteristic parameter information is subsequently generated into a corresponding digital human. Each digital human characteristic parameter information in the set of digital human characteristic parameter information can be multiple, i.e., batch generation of multiple digital humans at the same time.
[0036] As an example, each parameter information under each characteristic type that can be selected can be selected on a digital human characteristic parameter selection page to obtain the set of digital human characteristic parameter information.
[0037] Step 202, for each digital human characteristic parameter information, the following generation steps are performed: Step 2021, generating an initial digital human video corresponding to the above-mentioned digital human characteristic parameter information.
[0038] In some embodiments, the execution subject can generate an initial digital human video corresponding to the above-mentioned digital human characteristic parameter information. The initial digital human video can be a preliminary generated digital human video. The initial digital human video can be a video that still needs to be optimized by a digital human.
[0039] As an example, the execution subject can generate an initial digital human video corresponding to the digital human feature parameter information based on AIGC (Artificial Intelligence Generated Content) technology.
[0040] It should be noted that in the process of generating the initial digital human video, it is necessary to ensure that each frame of the digital human in the initial digital human video faces a unified user object, and the same object cannot have different digital human IDs.
[0041] Step 2022, optimizing the face feature content corresponding to the initial digital human video to obtain a digital human optimized video.
[0042] In some embodiments, the execution subject can optimize the face feature content corresponding to the initial digital human video to obtain a digital human optimized video. The face feature content can be the face content in the face region. The digital human optimized video can be a digital human video after fine optimization of the face content.
[0043] As an example, first, the execution subject can obtain an initial digital human frame sequence corresponding to the initial digital human video. Then, generate an optimization prompt word for fine optimization of the face content of each initial digital human frame in the initial digital human frame sequence. Finally, input the optimization prompt word and the initial digital human video into a pre-trained multi-modal large model to obtain a digital human optimized video.
[0044] In some optional implementations of some embodiments, the execution subject can optimize the face feature content corresponding to the initial digital human video to obtain a digital human optimized video, including the following steps: First, extract the audio feature representation information corresponding to the initial digital human video. The initial digital human video includes digital human audio. The digital human audio can be the audio content emitted by the digital human. The audio feature representation information can be the semantic content of the audio features corresponding to the digital human audio. In practice, the audio feature representation information can be a vector form of information.
[0045] As an example, the execution subject can extract the audio feature representation information corresponding to the digital human audio in the initial digital human video by pre-emphasis, framing and windowing, short-time Fourier transform, Mel filter bank processing, log energy calculation and discrete cosine transform.
[0046] Secondly, input the audio feature representation information into the pre-trained first face parameter mapping model to obtain digital face parameter information. The first face parameter mapping model can be a mapping model for determining the digital face parameter corresponding to the audio feature representation information. That is, the first face parameter mapping model can learn the mapping relationship between the audio feature representation information and the digital face parameter information. For example, the first face parameter mapping model can be a network model based on a Transformer structure. The digital face parameter information can be the parameter content corresponding to the digital face feature. For example, the digital face parameter information can include face contour information, facial feature positioning information, head posture information, skin texture information, and blinking mode.
[0047] Thirdly, the initial digital face image rendered by the digital face parameter information is subjected to face detail repair to obtain a digital face image. The face detail repair can be the repair of the face detail content. The face detail content corresponding to the digital face image is more refined than the face detail content corresponding to the initial digital face image.
[0048] As an example, the execution subject can use a GAN (Generative Adversarial Networks) repair model to repair the face details of the initial digital face image to obtain a digital face image.
[0049] Fourthly, the face feature content of the initial digital person video is optimized according to the digital face image to obtain a digital person optimization video.
[0050] As an example, first, the initial digital person frame sequence corresponding to the initial digital person video is determined. Then, for each initial digital person frame in the initial digital person frame sequence, the face feature content of the initial digital person frame is refined using a multi-modal large model according to the digital face image to obtain a digital person frame. Finally, the obtained digital person frame sequence is determined as a digital person optimization video.
[0051] Optionally, the first face parameter mapping model is trained by the following steps: In a first step, a target training dataset is obtained, wherein the target training data includes digital human audio and digital human video. The target training dataset is used to train the first initial face parameter mapping model. The first initial face parameter mapping model can be a first face parameter mapping model that has not yet completed training. The digital human audio in the target training data corresponds to audio content, and the digital human video corresponds to video content that is synchronized. In practice, a Stable Diffusion (SD) based audio-lip synchronization model (based on the wav2lip model, the SD structure is improved, and the richness of the lip shape is greatly improved) is used to match the lip movement in the digital human video with the digital human audio, and high-quality target training data can be obtained.
[0052] In a second step, for each target training data, the following generation steps are performed: Sub-step 1: Determine the high-quality speech representation information corresponding to the digital human audio included in the target training data. The high-quality speech representation information can represent high-quality audio semantic content. In practice, the high-quality speech representation information can be HuBERT feature corresponding feature information. The high-quality speech representation information is a highly structured and context-aware speech representation that maintains the semantic features of the audio, decouples the tone, facilitates subsequent tone replacement, and maintains the underlying acoustic details such as pitch and tone.
[0053] Sub-step 2: Extract the face parameter information corresponding to the digital human video included in the target training data.
[0054] As an example, the execution subject can perform 3DMM modeling on the digital human video to extract the parameterized geometry and texture parameters of the face, obtaining the face parameter information.
[0055] In a third step, the second initial face parameter mapping model is trained based on the obtained high-quality speech representation information set and face parameter information set, obtaining a second face parameter mapping model. The second initial face parameter mapping model can be a second face parameter mapping model that has not yet completed training. The second face parameter mapping model can be a mapping model that determines the digital human face parameters corresponding to the high-quality speech representation information. That is, the second face parameter mapping model can learn the mapping relationship between the high-quality speech representation information and the digital human face parameter information. For example, the second face parameter mapping model can be a network model based on a Transformer structure. The second face parameter mapping model can be a lightweight parameter mapping model.
[0056] As an example, the execution subject can take the high-quality speech representation information set as the input data set, the face parameter information set as the label, and perform model sequence on the second initial face parameter mapping model to obtain the second face parameter mapping model.
[0057] In the fourth step, the second face parameter mapping model is taken as the teacher model, the first initial face parameter mapping model is taken as the student model to be trained, and the first initial face parameter mapping model is trained to obtain the first face parameter mapping model.
[0058] Here, using the Teacher-student network, the training of the mel feature (i.e., the audio feature representation information) by minimizing the distillation loss using the output of the HuBERT feature as a constraint can make the lightweight student model achieve a performance close to that of the teacher model while meeting the deployment requirements of low latency and low resource consumption.
[0059] In step 2023, the action connection processing between each digital human action segment corresponding to the digital human optimized video is performed to obtain a digital human video.
[0060] In some embodiments, the execution subject can perform action connection processing between each digital human action segment corresponding to the digital human optimized video to obtain a digital human video. The action connection processing can be content optimization processing of the action switching content between each digital human action segment to improve the smoothness of the subsequent digital human video when performing action switching. The digital human action segment corresponds to the execution action of the digital human. That is, the digital human action segment is a video segment in which the digital human performs the corresponding action.
[0061] As an example, first, the execution subject can generate a processing prompt word for performing action connection processing on each digital human action segment corresponding to the digital human optimized video. Then, the processing prompt word and the digital human optimized video are input into the multi-modal large model to obtain a digital human video.
[0062] In some optional implementations of some embodiments, after step 202, the step further includes: For each digital human video in the obtained digital human video set, an adjustment step is performed: Sub-step 1: According to the digital human feature parameter information corresponding to the digital human video, the digital human video is quality audited to obtain an audit result. The audit result can be one of the following: an audit result that the quality of the digital human video passes the audit, and an audit result that the quality of the digital human video fails the audit.
[0063] As an example, first, the execution subject can extract each digital human feature information corresponding to each digital human feature from the digital human video. Then, the parameter value comparison is performed between each digital human feature information and the digital human feature parameter information, and the comparison result is obtained as the review result. In the case where the comparison result represents consistency, the review result representing the quality review pass is generated. In the case where the comparison result represents inconsistency, the review result representing the quality review fail is generated.
[0064] Sub-step 2, in response to the fact that the review result represents that the digital human video has quality problems, the digital human video is adjusted according to the review result and the digital human feature parameter information, and an adjusted digital human video is obtained.
[0065] As an example, the execution subject adjusts the digital human video according to the review result and the digital human feature parameter information using a large language model, and obtains an adjusted digital human video.
[0066] Sub-step 3, the adjusted digital human video is published.
[0067] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the digital human batch generation method of some embodiments of the present disclosure, the required multiple digital human videos can be efficiently and automatically batch generated. Specifically, the reason why the related digital human video is not batch generated is not efficient is that manual digital human production has the problems of low efficiency, long cycle, high cost, and difficulty in scaling, resulting in low efficiency of batch production. Based on this, the digital human batch generation method of some embodiments of the present disclosure first acquires the digital human feature parameter information set selected from the digital human feature parameter library for digital human batch generation, to obtain the parameter content for subsequent digital human batch production. Then, for each digital human feature parameter information, the following generation steps are performed: first, the initial digital human video corresponding to the digital human feature parameter information can be accurately generated. Second, the face feature content of the initial digital human video is optimized to obtain a digital human optimization video, which can realize fine processing of the face detail content and obtain a more accurate digital human video. Third, the action connection processing is performed between each digital human action segment of the digital human optimization video to obtain a digital human video, which can effectively avoid the unnatural problem of the digital human when changing actions. In summary, through the selection of digital human feature parameter information, the generation of initial digital human video, and multi-aspect optimization, the automatic processing of the digital production process is performed, which can realize the batch generation of digital human videos on the basis of ensuring production efficiency.
[0068] Further reference is made to Figure 3, shows flow 300 of some other embodiments of the digital human batch generation method according to the present disclosure. The digital human batch generation method comprises the following steps: Step 301, obtaining a set of digital human feature parameter information selected from a digital human feature parameter library for digital human batch generation.
[0069] Step 302, for each digital human feature parameter information, performing the following generation steps: Step 3021, generating an initial digital human video corresponding to the above-mentioned digital human feature parameter information.
[0070] Step 3022, optimizing the facial feature content corresponding to the above-mentioned initial digital human video to obtain a digital human optimized video.
[0071] Step 3023, filtering out a first digital human action segment corresponding to a listening state type from the above-mentioned each digital human action segment to obtain a first digital human action segment set.
[0072] In some embodiments, the execution subject (e.g. Figure 1 The electronic device 101 shown can filter out a first digital human action segment corresponding to a listening state type from the above-mentioned each digital human action segment to obtain a first digital human action segment set. The non-listening state type can be a type in which the digital human generates a fixed output (such as voice, animation) according to a preset program or text content, lacking real-time perception and dynamic interaction capability. The listening state type can be a type in which the digital human can perceive user input (such as voice, text, expression, etc.) in real time through voice recognition (ASR), natural language processing (NLP), etc. and generate dynamic responses based on the interaction mode. The first and last frames of the above-mentioned each digital human action segment are the same. Here, the first frame of the digital human image and the last frame of the digital human image of each digital human action segment are the same. In order to avoid unnatural performance of the digital human when switching actions.
[0073] Step 3024, for each first digital human action segment, performing the following replacement steps: Step 30241, in response to determining that the next action segment corresponding to the above-mentioned first digital human action segment is a segment of a non-listening state type, determining an action transition point corresponding to the above-mentioned first digital human action segment.
[0074] In some embodiments, in response to determining that the next action segment corresponding to the above-mentioned first digital human action segment is a segment of a non-listening state type, the execution subject can determine an action transition point corresponding to the above-mentioned first digital human action segment. The action transition point can be a time point for action transition. The next action segment corresponding to the above-mentioned first digital human action segment can be an action segment to be executed after the execution of the first digital human action segment.
[0075] As an example, the execution subject can take the frame time corresponding to the last key frame in the first digital human action segment as the action transition point.
[0076] Optionally, in response to determining that the first digital human action segment corresponds to a next action segment that is a non-listening state type segment, the execution subject can not adjust the digital human frame sequence between the first digital human action segment and the next action segment.
[0077] In some optional implementations of some embodiments, the execution subject can determine the action transition point corresponding to the first digital human action segment, including the following steps: First, for each digital human frame sub-sequence in the first digital human action segment, determine the similarity information between each digital human frame in the digital human frame sub-sequence and the first and last frames, to obtain a set of similarity information. The similarity information can be the similarity of image feature content. In practice, the similarity information can be based on the vector similarity between image feature vectors. The number of digital human frames in the digital human frame sub-sequence can be at least one. That is, the digital human frame sub-sequence can include one or more digital human frames.
[0078] Second, determine the digital human frame sub-sequence that corresponds to the set of similarity information satisfying the target similarity condition as the target digital human frame sub-sequence. The target similarity condition can be the digital human frame sub-sequence with the highest average similarity value in the set of similarity information.
[0079] Third, determine the action transition point according to the target digital human frame sub-sequence.
[0080] As an example, the execution subject can take the time point corresponding to the starting frame in the target digital human frame sub-sequence as the action transition point.
[0081] As another example, the execution subject can determine the time point corresponding to the last key frame in the target digital human frame sub-sequence as the action transition point.
[0082] Step 30242, using a video frame interpolation algorithm to generate a digital human frame sequence from the action transition point to the next action segment.
[0083] In some embodiments, the execution subject can use a video frame interpolation algorithm to generate a digital human frame sequence from the action transition point to the next action segment. The video frame interpolation algorithm can be an algorithm that performs digital human frame interpolation processing.
[0084] As an example, the execution subject can take the video frame corresponding to the action transition point as the starting frame, take the starting frame corresponding to the next action segment as the ending frame, use a video frame interpolation algorithm to generate and insert video frames, and obtain the digital human frame sequence.
[0085] In step 30243, the segment after the action transition point in the first digital human action segment is replaced by the digital human frame sequence, and an action replacement segment is obtained.
[0086] In some embodiments, the execution subject can replace the segment after the action transition point in the first digital human action segment with the digital human frame sequence to obtain an action replacement segment.
[0087] In step 3025, the digital human video is generated according to the obtained action replacement segments.
[0088] In some embodiments, the execution subject can generate the digital human video according to the obtained action replacement segments.
[0089] As an example, the execution subject can combine the action replacement segments and the un-replaced action segments in chronological order to obtain the digital human video.
[0090] In some embodiments, the specific implementation of steps 301, 3021-3022 and the technical effects brought by them can be referred to Figure 2 The steps 201, 2021-2022 in the corresponding embodiments will not be repeated here.
[0091] From Figure 3 It can be seen that, compared with Figure 2 the description of some embodiments corresponding to Figure 3 The flow 300 of the digital human batch generation method in some embodiments corresponding to the flow 200 of the digital human batch generation method in some embodiments can realize effective connection between action segments by performing digital human frame generation and insertion processing on the time period between the listening state type segment and the adjacent non-listening state type segment, so that the switching between the digital human action segments in the subsequent digital human video frames is more natural.
[0092] Further reference to Figure 4 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a digital human batch generation apparatus, which corresponds to the method embodiments shown in Figure 2 The digital human batch generation apparatus can be specifically applied to various electronic devices.
[0093] As Figure 4As shown, a digital human batch generation apparatus 400 includes an acquisition unit 401 and an execution unit 402. The acquisition unit 401 is configured to acquire a set of digital human characteristic parameter information selected from a digital human characteristic parameter library for digital human batch generation. The execution unit 402 is configured to, for each piece of digital human characteristic parameter information, perform the following generation steps: generate an initial digital human video corresponding to the digital human characteristic parameter information; optimize facial feature content corresponding to the initial digital human video to obtain a digital human optimized video; and perform motion connection processing between each digital human action segment corresponding to the digital human optimized video to obtain a digital human video.
[0094] In some optional implementations of some embodiments, the execution unit 402 can be further configured to: extract audio feature representation information corresponding to the initial digital human video; input the audio feature representation information into a pre-trained first facial parameter mapping model to obtain digital human facial parameter information; perform facial detail repair on an initial digital human facial image rendered from the digital human facial parameter information to obtain a digital human facial image; and optimize facial feature content corresponding to the initial digital human video according to the digital human facial image to obtain a digital human optimized video.
[0095] In some optional implementations of some embodiments, the first and last frames of each digital human action segment are the same, and the execution unit 402 can be further configured to: filter out a first digital human action segment corresponding to a listening state type from each digital human action segment to obtain a set of first digital human action segments; for each first digital human action segment, perform the following replacement steps: determine an action transition point of the first digital human action segment in response to determining that a next action segment corresponding to the first digital human action segment is a segment of a non-listening state type; generate a sequence of digital human frames from the action transition point to the next action segment using a video frame interpolation algorithm; replace a segment after the action transition point in the first digital human action segment with the sequence of digital human frames to obtain an action replacement segment; and generate the digital human video according to each action replacement segment obtained.
[0096] In some optional implementations of some embodiments, the execution unit 402 can be further configured to: for each digital human frame sub-sequence in the first digital human action segment, determine similarity information between each digital human frame in the digital human frame sub-sequence and the first and last frames to obtain a set of similarity information; determine a digital human frame sub-sequence corresponding to the set of similarity information that satisfies a target similarity condition as a target digital human frame sub-sequence; and determine the action transition point according to the target digital human frame sub-sequence.
[0097] In some optional implementations of some embodiments, the execution unit 402 can be further configured to, for each digital human video in the obtained set of digital human videos, perform an adjustment step of: performing quality review on the digital human video according to the digital human characteristic parameter information corresponding to the digital human video to obtain a review result; in response to the review result indicating that the digital human video has quality problems, performing video adjustment on the digital human video according to the review result and the digital human characteristic parameter information to obtain an adjusted digital human video; and performing video publishing on the adjusted digital human video.
[0098] In some optional implementations of some embodiments, the execution unit 402 can be further configured to: obtain a target training data set, wherein the target training data includes: digital human audio and digital human video; for each target training data, perform the following generation step: determine high-quality speech representation information corresponding to the digital human audio included in the target training data; extract face parameter information corresponding to the digital human video included in the target training data; perform model training on the second initial face parameter mapping model according to the obtained set of high-quality speech representation information and the set of face parameter information to obtain a second face parameter mapping model; use the second face parameter mapping model as a teacher model, use the first initial face parameter mapping model as a student model to be trained, and perform model training on the first initial face parameter mapping model to obtain a first face parameter mapping model.
[0099] It can be understood that the units described in the digital human batch generation apparatus 400 correspond to the respective steps in the method described with reference to Figure 2 Thus, the operations, features and advantages described above with respect to the method also apply to the digital human batch generation apparatus 400 and the units included therein, and will not be described here again.
[0100] Reference is made below to Figure 5 , which shows a structural schematic diagram of an electronic device (e.g., the electronic device 101 in Figure 1 adapted to implement some embodiments of the present disclosure. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.
[0101] As Figure 5As shown, the electronic device 500 can include a processing device (e.g., a central processor, a graphics processor, etc.) 501 that can perform various suitable actions and processes in accordance with programs stored in a read-only memory 502 or loaded from a storage device 508 into a random access memory 503. Various programs and data required by the electronic device 500 to operate are also stored in the random access memory 503. The processing device 501, the read-only memory 502, and the random access memory 503 are connected to each other by a bus 504. An input / output interface 505 is also connected to the bus 504.
[0102] Generally, the following devices can be connected to the input / output interface 505: input devices 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 509. The communication devices 509 can allow the electronic device 500 to communicate wirelessly or wired with other devices to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or present. More or fewer devices can alternatively be implemented or present. Figure 5 Each block shown in the flow diagrams can represent a device or multiple devices as needed.
[0103] In particular, processes described above with reference to the flow diagrams can be implemented as a computer software program according to some embodiments of the present disclosure. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flow diagrams. In some such embodiments, the computer program can be downloaded and installed from a network through the communication devices 509, or installed from the storage devices 508, or installed from the read-only memory 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the methods of some embodiments of the present disclosure are performed.
[0104] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0105] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.
[0106] The computer readable medium can be included in the electronic device; or can exist separately from the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to perform the following: obtaining a set of digital human characteristic parameter information selected from a digital human characteristic parameter library for batch generation of digital humans; for each digital human characteristic parameter information, performing the following generation steps: generating an initial digital human video corresponding to the digital human characteristic parameter information; optimizing facial feature content corresponding to the initial digital human video to obtain a digital human optimized video; and performing motion connection processing between each digital human action segment corresponding to the digital human optimized video to obtain a digital human video.
[0107] Computer program code for carrying out operations of some embodiments of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).
[0108] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified functions. It should also be noted that, in some alternative implementations, the functions noted in the blocks can occur in a different order than that noted in the figures. For example, two blocks noted in succession can in fact be executed substantially concurrently or in the opposite order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware-based systems and computer instructions.
[0109] The units described in some embodiments of the present disclosure can be implemented in the form of software, or can be implemented in the form of hardware. The described units can also be arranged in a processor, for example, can be described as: a processor includes an acquisition unit and an execution unit. Among them, the name of these units does not constitute a limitation to the unit itself in some cases, for example, the acquisition unit can also be described as: "a unit for acquiring a set of digital human feature parameter information selected from the digital human feature parameter library for batch generation of digital humans".
[0110] The functions described above in the present document can be performed at least in part by one or more hardware logic components. For example, non-limiting examples of exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on a chip (SOCs), complex programmable logic devices (CPLDs), etc.
[0111] Some embodiments of the present disclosure also provide a computer program product comprising a computer program which, when executed by a processor, implements any of the above-mentioned digital human batch generation methods.
[0112] The above description is merely some preferred embodiments of the present disclosure and a description of the principles of the technology used. Those skilled in the art should understand that the scope of the application involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features and the technical features disclosed in the embodiments of the present disclosure (but not limited to) having similar functions are replaced with each other to form a technical solution.
Claims
1. A method for mass generation of digital humans, comprising: Obtain the set of digital human feature parameters selected from the digital human feature parameter library for batch generation of digital humans; For each digital human feature parameter, perform the following generation steps: Generate an initial digital human video corresponding to the digital human feature parameter information; The facial feature content corresponding to the initial digital human video is optimized to obtain an optimized digital human video; The motion splicing process is performed between the various digital human motion segments corresponding to the optimized digital human video to obtain the digital human video.
2. The method according to claim 1, wherein, The optimization of the facial feature content corresponding to the initial digital human video to obtain an optimized digital human video includes: Extract the audio feature representation information corresponding to the initial digital human video; The audio feature representation information is input into a pre-trained first face parameter mapping model to obtain digital face parameter information; The initial digital face image rendered using the digital face parameter information is subjected to facial detail restoration to obtain a digital face image; Based on the digital human face image, the facial feature content corresponding to the initial digital human video is optimized to obtain an optimized digital human video.
3. The method according to claim 1, wherein, The first and last frames of each digital human motion segment are the same; as well as The step of performing motion splicing processing between the various digital human motion segments corresponding to the optimized digital human video to obtain the digital human video includes: The first digital human action segment corresponding to the listening state type is selected from each of the digital human action segments to obtain the first digital human action segment set; For each first digital human motion segment, perform the following replacement steps: In response to determining that the next action segment corresponding to the first digital human action segment is a non-listening state type segment, the action transition point corresponding to the first digital human action segment is determined; Using a video frame interpolation algorithm, a digital human frame sequence from the action transition point to the next action segment is generated; The segment after the action transition point in the first digital human action segment is replaced with the digital human frame sequence to obtain the action replacement segment; The digital human video is generated based on the obtained action replacement segments.
4. The method according to claim 3, wherein, Determining the action transition point corresponding to the first digital human action segment includes: For each digital human frame subsequence in the first digital human action segment, determine the similarity information between each digital human frame in the digital human frame subsequence and the first and last frames to obtain a similarity information set; The digital human frame subsequence that meets the target similarity condition and is selected from the first digital human action segment is determined as the target digital human frame sequence. The action transition point is determined based on the target digital human frame subsequence.
5. The method according to claim 1, wherein, The method further includes: For each digital human video in the obtained digital human video set, perform the adjustment steps: Based on the digital human feature parameter information corresponding to the digital human video, the digital human video is subjected to quality review to obtain the review result; In response to the audit result indicating that the digital human video has quality problems, the digital human video is adjusted according to the audit result and the digital human feature parameter information to obtain an adjusted digital human video; The adjusted digital human video is then published.
6. The method according to claim 2, wherein, The first facial parameter mapping model was trained through the following steps: Obtain the target training dataset, which includes: digital human audio and digital human video; For each target training data set, perform the following generation steps: The target training data includes high-quality speech representation information corresponding to the digital human audio. Extract facial parameter information corresponding to the digital human videos included in the target training data; Based on the obtained high-quality speech representation information set and face parameter information set, the second initial face parameter mapping model is trained to obtain the second face parameter mapping model. The second facial parameter mapping model is used as the teacher model, and the first initial facial parameter mapping model is used as the student model to be trained. The first initial facial parameter mapping model is trained to obtain the first facial parameter mapping model.
7. A device for mass-producing digital humans, comprising: The acquisition unit is configured to acquire a set of digital human feature parameter information selected from the digital human feature parameter library for batch generation of digital humans; The execution unit is configured to perform the following generation steps for each digital human feature parameter information: generate an initial digital human video corresponding to the digital human feature parameter information; optimize the facial feature content corresponding to the initial digital human video to obtain an optimized digital human video; and perform motion connection processing between the various digital human motion segments corresponding to the optimized digital human video to obtain a digital human video.
8. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.
9. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.
10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.