Multi-source single-graph digital human generation method and device, equipment and storage medium

Abstract

The invention provides a multi-source single-graph digital human generation method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a to-be-processed image, a driving audio and a driving video; performing face detection processing on the to-be-processed image to obtain a first face image; determining an audio driving result based on the first face image and the driving audio; performing face detection processing on the driving video, and determining a face video containing a second face image; and performing fusion processing based on the face video and the audio driving result to generate a face image video of the digital human. According to the embodiment of the invention, the defects that the face action and the speaking audio of the generated digital human are not matched and the accuracy is relatively low in the prior art are overcome, the single-image digital human with high matching degree between the audio and the face action can be generated by combining the to-be-processed image, the driving audio and the driving video, and the single-image digital human has diversity and flexibility.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for generating a multi-source single-image digital human. Background Art

[0002] In related technologies, methods for generating single-image digital humans can be divided into two categories according to the input driving source: one is to generate single-image digital humans based on audio driving. Although the audio-driven method can generate lip movements corresponding to the audio, due to the lack of movement information of other parts of the face, the overall movement of the face is uncontrollable during the generation, and the movement amplitude is relatively small.

[0003] The other type involves generating a single-image digital human based on video. This method can generate facial movements that match the video's movements, and the mouth movements are also generated based on the video's movements. However, if the audio input is inconsistent with the original video, there may be cases where the audio and mouth movements are inconsistent.

[0004] In actual applications, there may be situations where only audio is available, or there may be a need to specify a head movement template. Current technology makes it difficult to process these two input control sources at the same time, resulting in a mismatch between the generated digital human facial movements and the speaking audio, and low accuracy. Summary of the Invention

[0005] The present invention provides a multi-source single-image digital human generation method, device, equipment and storage medium, which are used to solve the defects of the existing technology that the facial movements and speaking audio of the digital human generated do not match and the accuracy is low. By combining the image to be processed, the driving audio and the driving video, a single-image digital human with a high degree of matching between audio and facial movements can be generated, and the single-image digital human has diversity and flexibility.

[0006] In a first aspect, the present invention provides a method for generating a digital human from a multi-source single image, comprising the following steps: Get the image to be processed, get the driving audio, and get the driving video; Performing face detection on the image to be processed to obtain a first face image; Determining an audio driving result based on the first facial image and the driving audio; Performing face detection processing on the driving video to determine a face video containing a second face image; Based on the face video and the audio driving result, fusion processing is performed to generate a face image video of a digital human; wherein the face in the face image video is the same as the face in the first face image, and the face movement in the face image video is the same as the face movement in the second face image.

[0007] Preferably, according to the multi-source single-image digital human generation method provided by the present invention, performing face detection processing on the image to be processed to obtain the first face image includes: Performing feature extraction processing on the image to be processed to obtain feature maps of different resolutions; In the process of top-down sampling and lateral connection of feature maps of different resolutions, upsampling the upper feature map so that the size of the upper feature map is the same as that of the lower feature map, and fusing the upper feature map with the lower feature map of the same size to obtain a semantically enhanced feature map; Based on the anchor boxes preset on the semantic enhancement feature map, predicting the confidence of the presence of a face in each anchor box and predicting the coordinate offset of each anchor box; Calculate the face bounding box coordinates based on the coordinate offset of each anchor box and the initial position and size information of the corresponding anchor box; Based on the confidence corresponding to each anchor frame, a face bounding box with the highest confidence score is screened out, and the first face image is cropped from the image to be processed according to the coordinates of the face bounding box with the highest confidence score.

[0008] Preferably, according to the multi-source single-image digital human generation method provided by the present invention, the audio driving result at least includes a video of a speaking face; The determining of an audio driving result based on the first facial image and the driving audio includes: performing three-dimensional face modeling processing on the first face image to obtain a three-dimensional face space, and extracting a three-dimensional identity coefficient of the first face image; Performing feature conversion processing on the driving audio to obtain audio features; Predicting a facial expression coefficient and a head posture coefficient based on the audio features and the three-dimensional identity coefficient; The facial expression coefficient and the head posture coefficient are mapped to the three-dimensional face space, and the facial expression coefficient and the head posture coefficient mapped to the three-dimensional face space are rendered using a three-dimensional face renderer to generate the speaking face video.

[0009] Preferably, according to a multi-source single-image digital human generation method provided by the present invention, the fusion processing based on the face video and the audio driving result to generate the digital human face image video includes: Extracting features from each frame of the speaking face video to generate a multi-scale feature map; Performing feature extraction on each frame of the facial video containing the second facial image to generate a feature map of the driving video; Fusing the multi-scale feature map with the feature map of the driving video, and adjusting feature weight values ​​in the fusion process in combination with an occlusion mask to generate a fused feature map; Perform feature decoding processing on the fused feature map to generate a face image video of the digital human.

[0010] Preferably, according to the multi-source single-image digital human generation method provided by the present invention, after the step of extracting features from each frame of the face video containing the second face image to generate a feature map of the driving video, the method includes: Calculating an optical flow field based on the feature map of the driving video to obtain motion information describing a face in the driving video; The motion information of the face in the driving video is migrated to the multi-scale feature map, so that the motion information of the face in the speaking face video is the same as the motion information of the face in the driving video.

[0011] Preferably, according to the multi-source single-image digital human generation method provided by the present invention, after the step of performing fusion processing based on the face video and the audio driving result to generate the face image video of the digital human, the method includes: The first face image in the image to be processed is replaced by face images with different face movements in the face image video to obtain the digital human with different face movements.

[0012] In a second aspect, the present invention further provides a multi-source single-image digital human generation device, comprising: An acquisition module is used to acquire images to be processed, drive audio, and drive video; A face detection module, configured to perform face detection on the image to be processed to obtain a first face image; A first determining module, configured to determine an audio driving result based on the first facial image and the driving audio; A second determination module is configured to perform face detection processing on the driving video to determine a face video containing a second face image; A generation module is used to perform fusion processing based on the face video and the audio driving result to generate a face image video of a digital human; wherein the face in the face image video is the same as the face in the first face image, and the face action in the face image video is the same as the face action in the second face image.

[0013] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multi-source single-image digital human generation method as described above is implemented.

[0014] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multi-source single-image digital human generation methods described above.

[0015] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described multi-source single-image digital human generation methods.

[0016] The present invention provides a multi-source single-image digital human generation method, device, equipment, and storage medium. The method comprises the following steps: obtaining an image to be processed, obtaining driving audio, and obtaining a driving video; performing face detection processing on the image to be processed to obtain a first facial image; determining an audio driving result based on the first facial image and the driving audio; performing face detection processing on the driving video to determine a facial video containing a second facial image; and performing fusion processing based on the facial video and the audio driving result to generate a digital human facial image video. The method is used to address the defects of the prior art in that the facial movements of digital humans generated do not match the speaking audio and are therefore of low accuracy. The method achieves this by combining the image to be processed, the driving audio, and the driving video to generate a single-image digital human with a high degree of matching between audio and facial movements, and the single-image digital human has diversity and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is one of the flow charts of the multi-source single-image digital human generation method provided by the present invention.

[0019] Figure 2 This is the second flow chart of the multi-source single-image digital human generation method provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of the multi-source single-image digital human generation device provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0023] The following combination Figure 1-Figure 4 The present invention describes a multi-source single-image digital human generation method, device, equipment and storage medium, which are used to solve the defects of the existing technology that the facial movements and speaking audio of the generated digital human do not match and the accuracy is low. By combining the image to be processed, the driving audio and the driving video, a single-image digital human with a high degree of matching between audio and facial movements can be generated, and the single-image digital human has diversity and flexibility.

[0024] Figure 1 This is one of the flow charts of a multi-source single-image digital human generation method provided by the present invention, such as Figure 1 As shown, the method may include but is not limited to steps S100 to S500: S100, obtaining an image to be processed, obtaining driving audio, and obtaining driving video; S200, performing face detection on the image to be processed to obtain a first face image; S300, determining an audio driving result based on the first facial image and the driving audio; S400, performing face detection processing on the driving video to determine a face video containing a second face image; S500, performing fusion processing based on the face video and the audio driving result to generate a face image video of a digital human; wherein the face in the face image video is the same as the face in the first face image, and the face action in the face image video is the same as the face action in the second face image.

[0025] The following combination Figure 1 and Figure 2 The following embodiments describe in detail a multi-source single-image digital human generation method provided by the present invention. In step S100 of some embodiments, an image to be processed, a driving audio, and a driving video are obtained.

[0026] It is understandable that the image to be processed is an image containing a human face, such as a static face image (such as a photo) uploaded by a user or obtained from a database, which serves as the basic identity feature for generating a digital person.

[0027] The driving audio is a piece of audio (such as voice conversation, song, etc.) uploaded by the user or obtained from the audio library, which is used to drive the digital human's mouth movements and expressions.

[0028] The driving video is a target face video obtained from user uploads or video libraries, which contains the target face's motion information (such as expression changes, head movements, etc.) and is used to control the digital human's motion details.

[0029] The technical effect achieved by this embodiment is: ensuring that the generated digital human inherits the identity characteristics of the source image, while having mouth movements synchronized with the driving audio and movement details consistent with the driving video.

[0030] In step S200 of some embodiments, face detection is performed on the image to be processed to obtain a first face image.

[0031] It is understandable that a face detection model (such as RetinaFace) can be used to perform face detection on the input image to be processed, locate the face area and crop the face part to obtain the first face image.

[0032] This embodiment can accurately extract the face area in the source image, ensuring that the generated digital human retains the identity features of the source image.

[0033] In some embodiments of the present invention, performing face detection on the image to be processed to obtain a first face image specifically includes: Performing feature extraction processing on the image to be processed to obtain feature maps of different resolutions; In the process of top-down sampling and lateral connection of feature maps of different resolutions, upsampling the upper feature map so that the size of the upper feature map is the same as that of the lower feature map, and fusing the upper feature map with the lower feature map of the same size to obtain a semantically enhanced feature map; Based on the anchor boxes preset on the semantic enhancement feature map, predicting the confidence of the presence of a face in each anchor box and predicting the coordinate offset of each anchor box; Calculate the face bounding box coordinates based on the coordinate offset of each anchor box and the initial position and size information of the corresponding anchor box; Based on the confidence corresponding to each anchor frame, a face bounding box with the highest confidence score is screened out, and the first face image is cropped from the image to be processed according to the coordinates of the face bounding box with the highest confidence score.

[0034] Furthermore, the specific implementation steps and technical effects of performing feature extraction processing on the image to be processed to obtain feature maps of different resolutions are as follows: 1. Input image preprocessing: resize the image to a fixed size (e.g., 800×800) and perform normalization (e.g., mean and variance normalization).

[0035] 2. Feature Pyramid FPN Construction: Use a backbone network (such as ResNet) to extract multi-scale feature maps. The backbone network usually contains multiple convolutional layers and outputs feature maps of different resolutions (such as C2, C3, C4, C5).

[0036] For example, C2 is the feature map of a shallower layer (higher resolution), and C5 is the feature map of a deeper layer (lower resolution).

[0037] 3. Feature map output: Obtain feature maps of different resolutions from the backbone network as input for subsequent processing.

[0038] Feature Pyramid Generation: Utilizing the Feature Pyramid Network (FPN) structure, the feature maps output by the backbone network are sampled and laterally connected from top to bottom. In this top-down process, the upper-layer feature maps are upsampled to the same size as the lower-layer feature maps and then fused with the lower-layer feature maps to enhance feature expression capabilities. This results in semantically rich feature maps of varying scales for subsequent face detection and key point location. This embodiment achieves the technical effect of extracting multi-scale feature maps through the backbone network, enabling the capture of semantic information at different levels (such as edges, textures, and high-level semantics). This also provides a foundation for subsequent feature enhancement and face detection.

[0039] Furthermore, in the process of top-down sampling and horizontal connection of feature maps of different resolutions, the upper-layer feature map is upsampled to make the size of the upper-layer feature map the same as that of the lower-layer feature map, and the upper-layer feature map with the same size is fused with the lower-layer feature map to obtain the semantic enhancement feature map. The specific implementation steps and technical effects of the implementation are as follows: 1. Top-down sampling: Starting from the deepest feature map (such as C5), gradually upsampling. For example, C5 is upsampled to the same size as C4 through transposed convolution or interpolation.

[0040] 2. Horizontal connection: Add the upsampled feature map to the original feature map of the corresponding layer (such as C4) element by element (i.e., feature fusion).

[0041] For example, the upsampled C5 is added to C4 to generate a new feature map P4.

[0042] 3. Repeated sampling and fusion: P4 is further upsampled and fused with C3 to generate P3, and so on, until semantically enhanced feature maps of all resolutions are generated (such as P2, P3, P4, P5).

[0043] The technical effect achieved by this embodiment is that high-level semantic information is integrated with low-level detail information through top-down sampling and lateral connections, enhancing the expressive power of the feature map. The resulting semantically enhanced feature map contains both high-level semantics and low-level detail, making it suitable for high-precision face detection.

[0044] Furthermore, based on the anchor boxes preset on the semantic enhancement feature map, the specific steps and technical effects of predicting the confidence of the presence of a face in each anchor box and predicting the coordinate offset of each anchor box are as follows: 1. Preset anchor boxes: Preset anchor boxes (e.g., k anchor boxes) of different scales and aspect ratios on each semantic enhancement feature map (e.g., P2, P3, P4, P5).

[0045] For example, anchor boxes are preset on P3 to cover faces of different sizes.

[0046] 2. Classification and regression prediction: Use the ClassHead network to classify each anchor box and predict its confidence level (such as softmax probability) of containing a face. Use the BoxHead network to regress each anchor box and predict its coordinate offset (such as , , , ).

[0047] Default box generation: A series of default boxes (Anchors) of different scales and aspect ratios are preset on the feature map. These default boxes cover the position and size range of possible faces in the image.

[0048] Feature Processing and Prediction: The feature maps generated from the feature pyramid are fed into the ClassHead and BoxHead networks. ClassHead predicts the presence of a face within each default box and the confidence level of the face; BoxHead predicts the coordinate offset of the face bounding box and the position offset of facial landmarks. This allows for refined classification and location prediction for each default box.

[0049] 3. Output prediction results: For each anchor box, output confidence and coordinate offset as input for subsequent decoding.

[0050] The technical effect of this embodiment is that by presetting anchor frames, faces of different scales and positions in an image can be covered. Classification and regression prediction can simultaneously detect the presence and precise location of a face.

[0051] Furthermore, based on the coordinate offset of each anchor box and the initial position and size information of the corresponding anchor box, the specific implementation steps and corresponding technical effects of calculating the coordinates of the face bounding box are as follows: 1. Coordinate decoding: coordinate offset of each anchor frame ( , , , ) is decoded, and the calculation formula is as follows: in,( , ) is the center coordinate of the anchor box, ( , ) are the width and height of the anchor box.

[0052] 2. Calculate the coordinates of the upper left and lower right corners of the face bounding box based on the decoded center coordinates and width and height: This embodiment achieves the technical effect of converting the offset of the anchor box into the actual face bounding box coordinates through coordinate decoding, ensuring that the detected face bounding box accurately matches the actual position in the image.

[0053] Furthermore, based on the confidence score corresponding to each anchor frame, the face bounding box with the highest confidence score is screened out, and the first face image is cropped from the image to be processed according to the coordinates of the face bounding box with the highest confidence score. The specific implementation steps and corresponding technical effects are as follows: 1. Confidence screening: Sort the confidence of all anchor boxes and filter out the anchor box with the highest confidence score (that is, the box most likely to contain a face).

[0054] 2. Non-Maximum Suppression (NMS): Non-maximum suppression is performed on the selected high-confidence anchor boxes, removing excessively overlapping boxes and retaining a single optimal face bounding box. Since multiple default boxes may predict the same face, this can result in multiple overlapping bounding boxes. The non-maximum suppression algorithm selects the highest-scoring bounding boxes within a given area, removing redundant boxes and retaining the most accurate face detection results.

[0055] 3. Face image cropping: According to the coordinates of the optimal face bounding box ( , , , ), cropping the facial region from the image to be processed to obtain a first facial image. This embodiment achieves the following technical effects: Through confidence filtering and non-maximum suppression, the final detected facial bounding box is ensured to be unique and optimal. The cropped facial image is precisely aligned, preserving the complete facial region, providing high-quality input for subsequent processing (such as audio-driven and motion transfer).

[0056] Through the above steps, we can achieve face detection based on RetinaFace and accurately crop the first face image from the image to be processed. The core technical effects include: 1. Multi-scale feature extraction: Extract feature maps of different resolutions through FPN to capture rich semantic information.

[0057] 2. Feature enhancement: Fusion of high-level semantics and low-level details through top-down sampling and lateral connections.

[0058] 3. Anchor box prediction: predict the confidence and bounding box coordinates of the face through ClassHead and BoxHead.

[0059] 4. Accurate decoding: Generate a unique and optimal face bounding box through coordinate decoding and non-maximum suppression.

[0060] 5. Quality cropping: Accurately crop the aligned face image from the image to be processed, providing a basis for subsequent processing.

[0061] In step S300 of some embodiments, an audio driving result is determined based on the first facial image and the driving audio.

[0062] It is understandable that the specific implementation steps may include: 1. Audio feature extraction: pre-processing the driving audio and extracting audio features (such as Mel-frequency cepstral coefficients MFCC, spectrogram, etc.).

[0063] 2. Audio-driven model to generate mouth movements: Use an audio-driven model (such as SadTalker) to convert audio features into mouth movement parameters to generate a preliminary speaking face video.

[0064] 3. 3D Face Reconstruction: The generated mouth movement parameters are combined with the 3D face model of the source image to generate a 3D face animation synchronized with the audio. The technical effect of this embodiment is that mouth movements synchronized with the driving audio are generated, ensuring that the digital human's mouth shape is consistent with the audio content. Through 3D face reconstruction, the naturalness and realism of mouth movements are enhanced.

[0065] In some embodiments of the present invention, the audio-driven result includes at least a video of a speaking person's face; The determining of an audio driving result based on the first facial image and the driving audio includes: performing three-dimensional face modeling processing on the first face image to obtain a three-dimensional face space, and extracting a three-dimensional identity coefficient of the first face image; Performing feature conversion processing on the driving audio to obtain audio features; Predicting a facial expression coefficient and a head posture coefficient based on the audio features and the three-dimensional identity coefficient; The facial expression coefficient and the head posture coefficient are mapped to the three-dimensional face space, and the facial expression coefficient and the head posture coefficient mapped to the three-dimensional face space are rendered using a three-dimensional face renderer to generate the speaking face video.

[0066] It is understandable that the specific implementation steps and technical effects achieved by performing three-dimensional face modeling on the first face image to obtain a three-dimensional face space and extracting the three-dimensional identity coefficient of the first face image are as follows: 1. 3D face modeling: Use a 3D face reconstruction model (such as Deep3DFace) to perform three-dimensional modeling on the first face image to generate a 3D face model.

[0067] Use key point detection (such as Dlib or Fan2DLib) to extract key points of facial features (such as eyes, nose, mouth, etc.) and fit a 3D face model.

[0068] 2. Extract three-dimensional identity coefficient: Parameterize the generated 3D face model and extract its three-dimensional identity coefficient (β) to represent the identity characteristics of the face (such as facial features, face shape, etc.).

[0069] These coefficients are usually based on the 3DMM (3D Morphable Model) model and are used to describe the geometric features of the face.

[0070] The technical effect achieved by this embodiment is: generating a high-precision 3D face model, providing a basis for subsequent facial expression and head posture driving.

[0071] The extracted 3D identity coefficients retain the identity features of the source image, ensuring that the generated digital human is consistent with the input face.

[0072] Furthermore, the driving audio is subjected to feature conversion processing to obtain audio features. The specific implementation steps and technical effects achieved are as follows: 1. Audio preprocessing: Perform preprocessing operations such as sampling rate unification and normalization on the driving audio.

[0073] 2. Audio feature extraction: Use audio feature extraction methods (such as MFCC, spectrogram, Hubert, etc.) to convert audio into feature vectors.

[0074] For example, extracting Mel-Frequency Cepstral Coefficients (MFCC) of audio or generating audio features using a pre-trained audio encoder such as Hubert.

[0075] 3. Audio feature serialization: Arrange audio features in chronological order to generate a feature sequence that matches the audio duration.

[0076] The technical effect achieved by this embodiment is: audio is converted into feature vectors suitable for model processing, providing input for subsequent audio driving. The extracted audio features can capture information such as the rhythm and intonation of the speech, providing a basis for mouth movement generation.

[0077] The specific implementation steps and technical effects achieved by predicting facial expression coefficients and head posture coefficients based on the audio features and the three-dimensional identity coefficients are as follows: 1. Audio-driven model prediction: Use the audio-driven module of the SadTalker model, taking audio features and three-dimensional identity coefficients as input.

[0078] The model predicts facial expression coefficients (such as lip opening and closing, eyebrow movements, etc.) and head posture coefficients (such as head rotation, tilt, etc.) synchronized with the audio by explicitly modeling the relationship between audio and facial expression coefficients and head posture coefficients.

[0079] 2. Coefficient Generation: Facial expression coefficients and head pose coefficients are typically expressed as parameters of a 3DMM model and are used to control the expressions and head pose of the 3D face model. This embodiment achieves the following technical benefits: The predicted facial expression coefficients and head pose coefficients are highly synchronized with the audio content, ensuring that the generated digital human's mouth movements are consistent with the audio. By explicitly modeling the relationship between audio and motion coefficients, more realistic facial expressions and head movements are generated.

[0080] The facial expression coefficients and the head posture coefficients are mapped to the three-dimensional face space, and the facial expression coefficients and the head posture coefficients mapped to the three-dimensional face space are rendered using a three-dimensional face renderer to generate the speaking face video. The specific implementation steps and technical effects achieved are as follows: 1. Coefficient mapping: Map the predicted facial expression coefficients and head posture coefficients into the three-dimensional face space and update the parameters of the 3D face model.

[0081] For example, facial expression coefficients are applied to the vertex coordinates of a 3D face model to generate new expressions.

[0082] 2. 3D face rendering: Use a 3D face renderer (such as FaceRenderer) to render the updated 3D face model.

[0083] The renderer generates a face image for each frame based on the mapped coefficients, including changes in facial expressions and head postures.

[0084] 3. Video synthesis: Arrange the rendered image sequence in chronological order to generate the final speaking face video.

[0085] The technical effect achieved by this embodiment is that in the generated speaking face video, the digital human's mouth movements are highly consistent with the audio content, and at the same time have natural facial expressions and head postures.

[0086] Through 3D face rendering, the generated video is ensured to be realistic and smooth, avoiding the artifacts and unnaturalness problems in 2D generation methods.

[0087] In some embodiments of the present invention, the steps of model optimization and effect improvement are also included: 1. ExpNet learns facial expressions: Use the ExpNet model to learn accurate facial expressions and ensure that the generated expressions are more natural.

[0088] 2. PoseVAE modeling head movement: Use the PoseVAE model to model head posture and generate different styles of head movements.

[0089] 3. Unsupervised 3D keypoint space mapping: Map the generated 3D motion coefficients to the unsupervised 3D keypoint space to further improve the rendering effect.

[0090] The technical effect of this embodiment is that the ExpNet and PoseVAE models generate more realistic facial expressions and head movements. Unsupervised 3D keypoint spatial mapping ensures that the generated video is more accurate in details.

[0091] Through the above steps, we can realize the audio driver based on the SadTalker model and generate a speaking face video corresponding to the audio content. The core technical effects include at least but not limited to: 1. 3D face modeling: retains the identity features of the source image and provides a basis for subsequent driving.

[0092] 2. Audio feature extraction: Capture the rhythm and intonation information of the audio to provide input for mouth movement generation.

[0093] 3. Explicit modeling: The SadTalker model explicitly models the relationship between audio and motion coefficients to generate realistic facial expressions and head postures.

[0094] 4. 3D rendering: Generate high-quality talking face videos through 3D face rendering to ensure authenticity and smoothness.

[0095] In step S400 of some embodiments, face detection processing is performed on the driving video to determine a face video containing a second face image.

[0096] It is understood that the specific implementation steps may include but are not limited to the following steps: 1. Face detection and tracking: Use a face detection model (such as RetinaFace) to detect faces in each frame of the driving video and locate the target face area.

[0097] 2. Key point extraction and motion estimation: Extract the key points of the target face's facial features and calculate the motion information between each frame (such as expression changes, head rotation, etc.) through the optical flow estimation algorithm.

[0098] 3. Motion parameter extraction: Parameterize the motion information of the target face (such as expression coefficient and head posture) to generate a motion feature sequence.

[0099] The technical effects achieved by this embodiment are: accurate extraction of target face motion information from the driving video, providing a motion reference for subsequent fusion. Through optical flow estimation, the generated digital human's motion continuity and naturalness are ensured.

[0100] In step S500 of some embodiments, a fusion process is performed based on the face video and the audio driving result to generate a face image video of a digital human; wherein the face in the face image video is the same as the face in the first face image, and the face action in the face image video is the same as the face action in the second face image.

[0101] It is understood that, in some embodiments, the specific execution steps may include but are not limited to the following steps: 1. Feature extraction and fusion: Use the feature extraction module of TPSM (such as ResNet) to extract feature maps of audio driving results and driving video respectively.

[0102] The spatial attention mechanism is used to generate an occlusion mask to distinguish between visible and invisible areas and fuse their features.

[0103] 2. Motion transfer: Use the temporal pyramid module of TPSM to perform temporal modeling on the motion information of the driving video and transfer the motion parameters of the target face to the audio driving results.

[0104] 3. Image generation and optimization: The fused feature map is decoded into an RGB image sequence through a deconvolution network.

[0105] The generated effect is optimized using perceptual loss, optical flow loss, identity loss, and occlusion loss to ensure that the digital human retains the identity features of the source image while also having the motion details of the target face.

[0106] 4. Video synthesis: Arrange the generated image sequence in chronological order to form the final digital human speaking video.

[0107] The technical effect achieved by this embodiment is that the generated digital human retains the identity characteristics of the source image (such as facial features and skin color), while also having mouth movements synchronized with the driving audio and motion details (such as facial expressions and head movements) consistent with the driving video. Furthermore, through the TPSM's spatial attention mechanism and temporal pyramid modeling, the generated video maintains high fidelity even in occluded areas, and its motion is smooth and natural.

[0108] In some embodiments of the present invention, the step of performing fusion processing based on the face video and the audio driving result to generate a face image video of a digital human includes: Extracting features from each frame of the speaking face video to generate a multi-scale feature map; Performing feature extraction on each frame of the facial video containing the second facial image to generate a feature map of the driving video; Fusing the multi-scale feature map with the feature map of the driving video, and adjusting feature weight values ​​in the fusion process in combination with an occlusion mask to generate a fused feature map; Perform feature decoding processing on the fused feature map to generate a face image video of the digital human.

[0109] It is understandable that the specific implementation steps and technical effects of extracting features from each frame of the speaking face video and generating a multi-scale feature map are as follows: 1. Video frame extraction: Decompose the audio-driven speaking face video into a frame-by-frame image sequence.

[0110] 2. Multi-scale feature extraction: Use a convolutional neural network (such as ResNet or VGG) to extract features from each image frame and generate a multi-scale feature map. The multi-scale feature map includes feature layers of different resolutions and is used to capture detailed facial information (such as facial features and texture).

[0111] 3. Feature map storage: The multi-scale feature maps of each frame are stored as a sequence to provide input for subsequent fusion.

[0112] The technical effects achieved by this embodiment are as follows: the generated multi-scale feature map can preserve the detailed information of the speaking face video, providing a basis for subsequent feature fusion with the driving video. Multi-scale feature extraction ensures the integrity of facial details and avoids feature loss.

[0113] Furthermore, the specific implementation steps and technical effects of performing feature extraction on each frame of the face video containing the second face image to generate a feature map of the driving video are as follows: 1. Driving video frame extraction: Decompose the driving video containing the second face image into a frame-by-frame image sequence.

[0114] 2. Driving video feature extraction: Use the same feature extraction network (such as ResNet or VGG) to extract features from each frame of the driving video and generate a feature map of the driving video.

[0115] 3. Feature map storage: Store the feature map of each frame of the driving video as a sequence to provide input for subsequent fusion.

[0116] The technical effects achieved by this embodiment are as follows: the generated driving video feature map can capture facial motion information (such as head posture and expression changes) in the driving video. By fusing it with the feature map of the audio driving result, more precise motion control can be achieved.

[0117] Furthermore, the multi-scale feature map and the feature map of the driving video are fused, and the feature weight values ​​in the fusion process are adjusted in combination with the occlusion mask to generate the fused feature map. The specific implementation steps and technical effects achieved are as follows: 1. Feature map alignment: The multi-scale feature maps of the speaking face video are spatiotemporally aligned with the feature maps of the driving video to ensure that they match at the same scale and timeline.

[0118] 2. Occlusion mask generation: Use the TPSM model to generate multi-resolution occlusion masks to identify occluded areas in the driving video (such as hair, background, etc.).

[0119] 3. Feature fusion: Fuse the multi-scale feature map with the feature map of the driving video, and adjust the fusion weight based on the occlusion mask: In unoccluded areas, the feature map of the driving video is preferentially used to preserve the motion information of the driving video.

[0120] In the occluded area, the feature map of the speaking face video is retained to ensure the consistency of identity features.

[0121] 4. Fusion feature map generation: Generate a fusion feature map through weighted fusion, while retaining the identity features of the audio-driven results and the action features of the driven video.

[0122] This embodiment achieves the following technical effects: The fused feature map combines the identity features of the audio-driven results with the motion features of the driving video, achieving more natural facial motion control. Furthermore, the use of occlusion masks ensures that the features of occluded areas are not destroyed, enhancing the realism of the generated video.

[0123] Furthermore, the specific implementation steps and technical effects of performing feature decoding processing on the fused feature map to generate the face image video of the digital human are as follows: 1. Feature decoding: Use a decoder network (such as FPN or U-Net) to decode the fused feature map and generate frame-by-frame face images.

[0124] The decoder is used to convert the fused feature map into an RGB image while preserving identity features and action information.

[0125] 2. Video synthesis: Arrange the decoded frame-by-frame facial images in chronological order to generate the final digital human face image video.

[0126] 3. Post-processing optimization: Smooth the generated video (such as optical flow alignment, inter-frame interpolation, etc.) to ensure the smoothness and naturalness of the video.

[0127] The technical effects achieved by this embodiment are as follows: In the generated digital human face image video, the face is consistent with the identity in the first face image, and the facial movements are consistent with the movements in the second face image. Through feature decoding and post-processing optimization, the generated video has high fidelity and smoothness.

[0128] In some embodiments, the specific steps of fitting the driven facial result (face image video) into the original input image include: 1. Face region extraction: Perform face detection on the original input image (such as using RetinaFace) and extract the face region.

[0129] 2. Face replacement: Replace the face area in the original input image with the driven face image, while retaining the rest of the body.

[0130] 3. Image fusion: Perform edge smoothing and color correction on the replaced image to ensure the natural fusion of the driven face and the original image.

[0131] The technical effect of this embodiment is that the generated digital human retains the body parts of the original input image, while the facial movements are consistent with the driving video. Through face replacement and image fusion, the generation of a full-body digital human is achieved.

[0132] Through the above steps, it is possible to generate a digital human face image video based on the fusion of audio-driven results and video-driven results. The core technical effects include at least the following: Multi-scale feature extraction: Preserving identity and detail information for audio-driven results.

[0133] Driving video feature fusion: Achieving precise motion control while preserving identity features through the TPSM model.

[0134] Occlusion mask optimization: Ensures that the features of the occluded area are not destroyed, improving realism.

[0135] Feature decoding and video synthesis: Generate high-fidelity digital human face image videos.

[0136] Face replacement and fusion: Fit the driving results to the original input image to generate a full-body digital human.

[0137] In some embodiments of the present invention, after the step of extracting features from each frame of the face video containing the second face image to generate a feature map of the driving video, the method includes: Calculating an optical flow field based on the feature map of the driving video to obtain motion information describing a face in the driving video; The motion information of the face in the driving video is migrated to the multi-scale feature map, so that the motion information of the face in the speaking face video is the same as the motion information of the face in the driving video.

[0138] It can be understood that the specific steps for calculating the optical flow field based on the feature map of the driving video and obtaining the motion information describing the face in the driving video are as follows: 1. Optical flow field calculation step: Use an optical flow estimation algorithm (such as PWC-Net, RAFT or traditional Lucas-Kanade method) to calculate the optical flow field of the feature map sequence of the driving video.

[0139] The optical flow field describes the pixel motion vectors (such as translation, rotation, etc.) of the face between adjacent frames in the driving video.

[0140] 2. Motion information extraction step: Extract key motion information from the optical flow field, such as the displacement of facial key points, head posture changes, etc.

[0141] The motion information is represented as motion vectors or deformation matrices for subsequent migration.

[0142] The technical effect achieved by this embodiment is that the optical flow field accurately describes the motion trajectory of the face in the driving video, providing a foundation for subsequent motion migration. Through optical flow field calculation, subtle movements in the driving video (such as blinking and lip opening and closing) can be captured, ensuring the integrity of the motion information.

[0143] Furthermore, the specific steps of migrating the motion information of the face in the driving video to the multi-scale feature map include: 1. Motion information mapping step: Map the motion information of the driving video (such as motion vectors or deformation matrices) to the multi-scale feature map of the speaking face video.

[0144] The motion information of the driving video is applied to the feature map of the speaking face video through affine transformation or optical flow alignment.

[0145] 2. Feature map deformation step: Based on the mapped motion information, the multi-scale feature map of the speaking face video is deformed so that its motion information is consistent with the driving video.

[0146] For example, the head rotation angle in the driving video is applied to the feature map of the talking face video.

[0147] 3. Motion consistency optimization step: Optimize the deformed feature map through loss functions (such as optical flow loss and feature consistency loss) to ensure that the motion information of the speaking face video is consistent with the driving video.

[0148] The technical effect achieved by this embodiment is that the facial motion information in the speaking face video is completely consistent with the driving video, achieving accurate motion migration.

[0149] Through feature map deformation and optimization, distortion and artifacts in the motion migration process are avoided, ensuring the naturalness of the generated video.

[0150] Furthermore, the specific steps of adjusting the feature weight values ​​in the fusion process in combination with the occlusion mask include: 1. Occlusion region detection step: Use the TPSM model to generate a multi-resolution occlusion mask to identify the occluded areas (such as hair, background, etc.) in the driving video.

[0151] 2. Feature weight adjustment step: During the fusion process, the feature weight is adjusted according to the occlusion mask: In unoccluded areas, the feature map of the driving video is preferentially used to preserve motion information.

[0152] In the occluded area, the feature map of the speaking face video is retained to ensure the consistency of identity features.

[0153] 3. Fusion feature map generation step: Generate a fusion feature map through weighted fusion, while retaining the motion information of the driving video and the identity features of the speaking face video.

[0154] The technical effect achieved by this embodiment is that the use of occlusion masks ensures that the features of occluded areas are not destroyed, improving the realism of the generated video. By adjusting feature weights, a balance is achieved between motion information and identity features, avoiding over-reliance on driving video or audio to drive the results.

[0155] Furthermore, the specific steps of performing feature decoding processing on the fused feature map to generate a digital human face image video include: 1. Feature decoding step: Use a decoder network (such as FPN or U-Net) to decode the fused feature map to generate frame-by-frame face images.

[0156] The decoder converts the fused feature map into an RGB image while preserving identity and motion information.

[0157] 2. Video synthesis step: Arrange the decoded frame-by-frame facial images in chronological order to generate the final digital human face image video.

[0158] 3. Post-processing optimization step: Smooth the generated video (such as optical flow alignment, inter-frame interpolation, etc.) to ensure the smoothness and naturalness of the video.

[0159] The technical effect achieved by this embodiment is that the facial motion of the generated digital human face video is consistent with the driving video, while retaining the identity characteristics of the speaking face video. Through feature decoding and post-processing optimization, the generated video has high fidelity and smoothness.

[0160] Through the above steps, the function of calculating the optical flow field based on the driving video feature map and migrating the motion information to the speaking face video can be realized. The core technical effects include: Precise motion migration: Through optical flow field calculation and feature map deformation, the motion information of the driving video is accurately migrated to the speaking face video. Identity feature retention: Through occlusion mask and feature weight adjustment, the identity features of the generated digital human are ensured to be consistent with the input. Improved realism: Through multi-resolution occlusion mask and feature fusion, artifacts and distortion in the motion migration process are avoided. Video smoothness: Through feature decoding and post-processing optimization, the generated digital human video has high fidelity and smoothness.

[0161] In some embodiments of the present invention, after the step of performing fusion processing based on the face video and the audio driving result to generate a face image video of a digital human, the method includes: The first face image in the image to be processed is replaced by face images with different face movements in the face image video to obtain the digital human with different face movements.

[0162] It is understood that the specific implementation steps may include but are not limited to the following: 1. Face detection and extraction step of the image to be processed: Use a model such as RetinaFace to perform face detection on the image to be processed and locate the area of ​​the first face image. Extract the first face area in the image to be processed, while retaining the rest of the body.

[0163] 2. Face image video frame extraction step: Decompose the generated face image video into a frame-by-frame image sequence, where each frame corresponds to a different facial action (such as speaking, nodding, shaking head, etc.).

[0164] 3. Face action replacement step: Replace the first face region in the image to be processed with each frame of the face image in the face image video: For each frame, the driven face image (including specific actions) is spliced ​​with the body part of the image to be processed.

[0165] Keep the background and body parts of the image to be processed and only replace the face area.

[0166] 4. Multi-action digital human generation step: By traversing all frames of the face image video, a sequence of digital human images with different facial actions is generated.

[0167] The technical effects of this embodiment include at least: the generated digital human retains the identity features and body parts of the processed image, and the facial movements are consistent with those in the facial image video. By replacing the image frame by frame, a variety of movements can be achieved (such as speaking, nodding, shaking the head, etc.).

[0168] In some embodiments of the present invention, the step of generating a video based on a sequence of digital human images with different facial movements is also included. The specific implementation steps may include: 1. Image sequence arrangement step: Arrange the generated digital human image sequence in chronological order to form a video frame sequence.

[0169] 2. Video synthesis step: Encode the image sequence into a video format (such as MP4) to generate digital human videos with different facial movements.

[0170] 3. Smoothing Processing Step: The generated video is smoothed (e.g., using optical flow alignment and interframe interpolation) to ensure the video's smoothness and naturalness. The technical effect of this embodiment is that the facial movements in the generated digital human video are consistent with those in the facial image video, while preserving the identity features and body parts of the processed image. Furthermore, the smoothing process enhances the video's smoothness and naturalness.

[0171] In some embodiments of the present invention, a multi-source single-image digital human generation method further includes: optimizing the replacement of digital human images based on different facial movements, including but not limited to the following steps: 1. Edge smoothing and color correction steps: Smooth the edges of the replaced digital human image to ensure a natural transition between the face and body. Perform color correction to ensure the color of the replaced face is consistent with the original image.

[0172] 2. Action consistency check step: Check whether the actions in the replaced digital human image sequence are coherent to ensure smooth transition of the actions.

[0173] 3. Multi-resolution processing steps: Perform resolution alignment and scaling processing on the processed images and facial image videos of different resolutions to ensure the consistent quality of the generated digital human images.

[0174] The technical effects achieved by this embodiment are: through edge smoothing and color correction, the replaced digital human image is more natural, avoiding obvious artifacts. Motion consistency checking ensures the smoothness of the digital human video. Multi-resolution processing improves the quality of the generated digital human image, adapting to inputs of different resolutions.

[0175] Through the above steps, it is possible to replace the first face image in the image to be processed based on the different face movements in the face image video, and generate a digital human with different face movements. The core technical effects include but are not limited to: 1. Identity Preservation: The generated digital human retains the identity and body parts of the processed image. 2. Action Diversification: By replacing each frame in the face image video, a variety of action expressions are achieved. 3. Natural Blending: Through edge smoothing, color correction, and multi-resolution processing, the replaced digital human image is ensured to be natural and high-quality. 4. Video Generation: The replaced image sequence is synthesized into a smooth digital human video, suitable for a variety of application scenarios (such as virtual idols and digital human live broadcasts).

[0176] The present invention provides a multi-source single-image digital human generation method, device, equipment, and storage medium. The method comprises the following steps: obtaining an image to be processed, obtaining driving audio, and obtaining a driving video; performing face detection processing on the image to be processed to obtain a first facial image; determining an audio driving result based on the first facial image and the driving audio; performing face detection processing on the driving video to determine a facial video containing a second facial image; and performing fusion processing based on the facial video and the audio driving result to generate a digital human facial image video. The method is used to address the defects of the prior art in that the facial movements of digital humans generated do not match the speaking audio and are therefore of low accuracy. The method achieves this by combining the image to be processed, the driving audio, and the driving video to generate a single-image digital human with a high degree of matching between audio and facial movements, and the single-image digital human has diversity and flexibility.

[0177] The multi-source single-image digital human generating apparatus provided by the present invention is described below. The multi-source single-image digital human generating apparatus described below and the multi-source single-image digital human generating method described above can be referenced to each other.

[0178] like Figure 3 The figure shows a schematic diagram of the structure of the multi-source single-image digital human generation device provided by the present invention. The multi-source single-image digital human generation device includes the following modules: An acquisition module 310 is used to acquire an image to be processed, drive audio, and drive video; A face detection module 320 is configured to perform face detection on the image to be processed to obtain a first face image; A first determining module 330 is configured to determine an audio driving result based on the first facial image and the driving audio; A second determination module 340 is configured to perform face detection processing on the driving video to determine a face video containing a second face image; The generation module 350 is used to perform fusion processing based on the face video and the audio driving result to generate a face image video of a digital human; wherein the face in the face image video is the same as the face in the first face image, and the face action in the face image video is the same as the face action in the second face image.

[0179] Preferably, the multi-source single-image digital human generation device provided by the present invention is specifically used to perform feature extraction processing on the image to be processed to obtain feature maps of different resolutions; In the process of top-down sampling and lateral connection of feature maps of different resolutions, upsampling the upper feature map so that the size of the upper feature map is the same as that of the lower feature map, and fusing the upper feature map with the lower feature map of the same size to obtain a semantically enhanced feature map; Based on the anchor boxes preset on the semantic enhancement feature map, predicting the confidence of the presence of a face in each anchor box and predicting the coordinate offset of each anchor box; Calculate the face bounding box coordinates based on the coordinate offset of each anchor box and the initial position and size information of the corresponding anchor box; Based on the confidence corresponding to each anchor frame, a face bounding box with the highest confidence score is screened out, and the first face image is cropped from the image to be processed according to the coordinates of the face bounding box with the highest confidence score.

[0180] Preferably, the multi-source single-image digital human generation device provided by the present invention is specifically used for The audio driving result at least includes a video of a speaking face; The determining of an audio driving result based on the first facial image and the driving audio includes: performing three-dimensional face modeling processing on the first face image to obtain a three-dimensional face space, and extracting a three-dimensional identity coefficient of the first face image; Performing feature conversion processing on the driving audio to obtain audio features; Predicting a facial expression coefficient and a head posture coefficient based on the audio features and the three-dimensional identity coefficient; The facial expression coefficient and the head posture coefficient are mapped to the three-dimensional face space, and the facial expression coefficient and the head posture coefficient mapped to the three-dimensional face space are rendered using a three-dimensional face renderer to generate the speaking face video.

[0181] Preferably, the multi-source single-image digital human generation device provided by the present invention is specifically used to extract features from each frame of the speaking face video to generate a multi-scale feature map; Performing feature extraction on each frame of the facial video containing the second facial image to generate a feature map of the driving video; Fusing the multi-scale feature map with the feature map of the driving video, and adjusting feature weight values ​​in the fusion process in combination with an occlusion mask to generate a fused feature map; Perform feature decoding processing on the fused feature map to generate a face image video of the digital human.

[0182] Preferably, the multi-source single-image digital human generation device provided by the present invention is specifically used to calculate the optical flow field based on the feature map of the driving video to obtain motion information describing the human face in the driving video; The motion information of the face in the driving video is migrated to the multi-scale feature map, so that the motion information of the face in the speaking face video is the same as the motion information of the face in the driving video.

[0183] Preferably, the multi-source single-image digital human generation device provided by the present invention is specifically used to replace the first face image in the image to be processed based on face images with different face actions in the face image video to obtain the digital human with different face actions.

[0184] The present invention provides a multi-source single-image digital human generation method, device, equipment, and storage medium. The method comprises the following steps: obtaining an image to be processed, obtaining driving audio, and obtaining a driving video; performing face detection processing on the image to be processed to obtain a first facial image; determining an audio driving result based on the first facial image and the driving audio; performing face detection processing on the driving video to determine a facial video containing a second facial image; and performing fusion processing based on the facial video and the audio driving result to generate a digital human facial image video. The method is used to address the defects of the prior art in that the facial movements of digital humans generated do not match the speaking audio and are therefore of low accuracy. The method achieves this by combining the image to be processed, the driving audio, and the driving video to generate a single-image digital human with a high degree of matching between audio and facial movements, and the single-image digital human has diversity and flexibility.

[0185] Figure 4 An example of a physical structure diagram of an electronic device is shown below. Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communications bus 440. The processor 410 may call logic instructions in the memory 430 to execute a multi-source single-image digital human generation method, which includes: obtaining an image to be processed, obtaining driving audio, and obtaining a driving video; performing face detection on the image to be processed to obtain a first facial image; determining an audio driving result based on the first facial image and the driving audio; performing face detection on the driving video to determine a facial video containing a second facial image; and performing fusion processing based on the facial video and the audio driving result to generate a facial image video of the digital human; wherein the face in the facial image video is the same as the face in the first facial image, and the facial motion in the facial image video is the same as the facial motion in the second facial image.

[0186] Furthermore, the logic instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0187] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-source single-image digital human generation method provided by the above-mentioned methods, which includes: obtaining an image to be processed, obtaining driving audio, and obtaining a driving video; performing face detection processing on the image to be processed to obtain a first face image; determining an audio driving result based on the first face image and the driving audio; performing face detection processing on the driving video to determine a face video containing a second face image; performing fusion processing based on the face video and the audio driving result to generate a face image video of a digital human; wherein, the face in the face image video is the same as the face in the first face image, and the face action in the face image video is the same as the face action in the second face image.

[0188] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multi-source single-image digital human generation method provided by the above-mentioned methods, the method comprising: obtaining an image to be processed, obtaining driving audio, and obtaining a driving video; performing face detection processing on the image to be processed to obtain a first face image; determining an audio driving result based on the first face image and the driving audio; performing face detection processing on the driving video to determine a face video containing a second face image; performing fusion processing based on the face video and the audio driving result to generate a face image video of a digital human; wherein, the face in the face image video is the same as the face in the first face image, and the face action in the face image video is the same as the face action in the second face image.

[0189] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0190] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multi-source single-image digital human generation method, characterized in that: include: Get the image to be processed, get the driving audio, and get the driving video; Performing face detection on the image to be processed to obtain a first face image; Determining an audio driving result based on the first facial image and the driving audio; Performing face detection processing on the driving video to determine a face video containing a second face image; Based on the face video and the audio driving result, fusion processing is performed to generate a face image video of a digital human; wherein the face in the face image video is the same as the face in the first face image, and the face movement in the face image video is the same as the face movement in the second face image.

2. The multi-source single-image digital human generation method according to claim 1, characterized in that: The step of performing face detection on the image to be processed to obtain a first face image includes: Performing feature extraction processing on the image to be processed to obtain feature maps of different resolutions; In the process of top-down sampling and lateral connection of feature maps of different resolutions, upsampling the upper feature map so that the size of the upper feature map is the same as that of the lower feature map, and fusing the upper feature map with the lower feature map of the same size to obtain a semantically enhanced feature map; Based on the anchor boxes preset on the semantic enhancement feature map, predicting the confidence of the presence of a face in each anchor box and predicting the coordinate offset of each anchor box; Calculate the face bounding box coordinates based on the coordinate offset of each anchor box and the initial position and size information of the corresponding anchor box; Based on the confidence corresponding to each anchor frame, a face bounding box with the highest confidence score is screened out, and the first face image is cropped from the image to be processed according to the coordinates of the face bounding box with the highest confidence score.

3. The multi-source single-image digital human generation method according to claim 1, characterized in that: The audio driving result at least includes a video of a speaking face; The determining of an audio driving result based on the first facial image and the driving audio includes: performing three-dimensional face modeling processing on the first face image to obtain a three-dimensional face space, and extracting a three-dimensional identity coefficient of the first face image; Performing feature conversion processing on the driving audio to obtain audio features; Predicting a facial expression coefficient and a head posture coefficient based on the audio features and the three-dimensional identity coefficient; The facial expression coefficient and the head posture coefficient are mapped to the three-dimensional face space, and the facial expression coefficient and the head posture coefficient mapped to the three-dimensional face space are rendered using a three-dimensional face renderer to generate the speaking face video.

4. The multi-source single-image digital human generation method according to claim 3, characterized in that: The step of performing fusion processing based on the face video and the audio driving result to generate a face image video of a digital human includes: Extracting features from each frame of the speaking face video to generate a multi-scale feature map; Performing feature extraction on each frame of the facial video containing the second facial image to generate a feature map of the driving video; Fusing the multi-scale feature map with the feature map of the driving video, and adjusting feature weight values ​​in the fusion process in combination with an occlusion mask to generate a fused feature map; Perform feature decoding processing on the fused feature map to generate a face image video of the digital human.

5. The multi-source single-image digital human generation method according to claim 4, characterized in that: After the step of extracting features from each frame of the face video containing the second face image to generate a feature map of the driving video, the method includes: Calculating an optical flow field based on the feature map of the driving video to obtain motion information describing a face in the driving video; The motion information of the face in the driving video is migrated to the multi-scale feature map, so that the motion information of the face in the speaking face video is the same as the motion information of the face in the driving video.

6. The multi-source single-image digital human generation method according to claim 2, characterized in that: After the step of performing fusion processing based on the face video and the audio driving result to generate a face image video of a digital human, the method includes: The first face image in the image to be processed is replaced by face images with different face movements in the face image video to obtain the digital human with different face movements.

7. A multi-source single-image digital human generation device, characterized in that: include: An acquisition module is used to acquire images to be processed, drive audio, and drive video; A face detection module, configured to perform face detection on the image to be processed to obtain a first face image; A first determining module, configured to determine an audio driving result based on the first facial image and the driving audio; A second determination module is configured to perform face detection processing on the driving video to determine a face video containing a second face image; A generation module is used to perform fusion processing based on the face video and the audio driving result to generate a face image video of a digital human; wherein the face in the face image video is the same as the face in the first face image, and the face action in the face image video is the same as the face action in the second face image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the multi-source single-image digital human generation method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-source single-image digital human generation method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-source single-image digital human generation method according to any one of claims 1 to 6 is implemented.

Images