High-definition digital human video generation method and system for customizing real human image, storage medium and equipment

Abstract

The invention relates to the technical field of digital persons, in particular to a high-definition digital person video generation method for customizing a real person image, which comprises the following steps of: S1, acquiring audio and video data of a target person of which the real person image needs to be customized; s2, preprocessing the audio and video data; s3, customizing an exclusive voice model according to the audio data; s2, segmenting the audio data in the step S2, using a speech recognition model to recognize segmented audio as a label, and training to obtain a customized exclusive speech model; s4, carrying out real-person image fitting generation on the preprocessed audio and video data; s5, performing combined rendering and output of the audio and the real person dynamic video; and inputting a text, generating an audio according to the exclusive voice model and the input text, rendering a real person dynamic video at the same time, fusing the audio and the real person dynamic video through post-processing, and outputting the fused audio and the real person dynamic video to obtain a high-definition digital human video matched with the voice. According to the invention, low-cost customized digital human video generation is realized, and real human voice cloning is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of digital human, and in particular to a method, system, storage medium and device for generating high-definition digital human videos with customized real-life images. Background Art

[0002] The technology of generating digital human videos with real-life images has been widely used in the fields of live broadcasting, education and training. However, on the one hand, it is difficult for existing digital human video generation systems to generate high-definition and realistic digital human videos with real-life images at a low cost. On the other hand, the existing digital human video generation systems mostly use general generated voices that do not match the real-life image. Even if customized voice generation is used, the customized voice production process is often time-consuming and labor-intensive. Summary of the invention

[0003] The purpose of the present invention is to provide a method for generating high-definition digital human videos with customized real-life images, realize low-cost customized digital human video generation, and realize real-life voice cloning.

[0004] In order to achieve the above object, a method for generating a high-definition digital human video with a customized real-life image is provided, comprising the following steps:

[0005] S1. Collect audio and video data of the target person whose real-life image needs to be customized;

[0006] S2, preprocessing the audio and video data; the preprocessing includes segmenting the collected audio and video data to obtain the head data, torso data and background data of the portrait, then extracting the facial features including the blinking information, and finally extracting the audio data;

[0007] S3, customizing a dedicated voice model according to the audio data; first, segmenting the audio data in step S2, and using the voice recognition model to recognize the segmented audio as annotations, and then training to obtain a customized dedicated voice model;

[0008] S4, fitting and generating a real person image for the preprocessed audio and video data; training a 3D Gaussian Splatting model with the head data, torso data, background data and facial features of the portrait preprocessed in step S2 to model a real person image, and rendering a real person dynamic video with the 3D Gaussian Splatting model;

[0009] S5. Perform combined rendering and output of audio and real-person dynamic video; input text, generate audio based on the exclusive voice model and input text, render real-person dynamic video at the same time, and then fuse the audio and real-person dynamic video through post-processing and output to obtain a high-definition digital human video with matching voice.

[0010] Further, the step S1 comprises the following steps:

[0011] S1-1, recording a live video; using a fixed camera position, recording the audio and video data of the target person at the target viewing angle required for rendering; during the audio and video recording of the target person, the target person speaks according to a preset script, and records the spatial position information corresponding to the target spatial position when the target person pronounces different words;

[0012] S1-2, perform video editing; crop the recorded audio and video so that the area where the target person is located is centered, and modify the resolution and fix the frame rate.

[0013] Further, the step S2 specifically includes the following steps:

[0014] S2-1, performing image region segmentation on the audio and video data; separating the audio and video data obtained in step S1 into video frame images and audio data, and then using the face-parsing model to segment and obtain face data, torso data and background data of each frame image;

[0015] S2-2, extract facial features from the video frame images; use OpenFace to extract the AU45 encoding features containing blink information from each video frame image;

[0016] S2-3. Extract pronunciation features from audio data; use the Hubert model to extract the pronunciation features of the target person in the audio.

[0017] Further, the step S3 specifically includes the following steps:

[0018] S3-1: Segment and annotate the audio. Divide the audio into short audios of similar duration according to the silent time points of the audio, and then use FunASR to recognize and annotate the segmented speech content.

[0019] S3-2: Generate and train a voice model; use the data obtained in step S3-1 to fine-tune the pre-trained GPT-SoVITS model, complete the training, and obtain a dedicated voice model with customized timbre.

[0020] Further, the step S4 specifically includes the following steps:

[0021] Step S4-1: Static fitting training is performed when fitting the real person image, and the average static image of the customized image is fitted from the video image; the process uses multiple Gaussian ellipsoids to fit the static image, wherein each Gaussian ellipsoid is defined as follows:

[0022]

[0023] μ is the position vector of the ellipsoid space, ∑ -1The covariance matrix contains the rotation and scale information of the ellipsoid in space; SH is the spherical harmonic function, which contains the color information presented when the ellipsoid is observed from different angles in space; α is the opacity of the ellipsoid;

[0024] Through image data training, adjust μ,∑ in the ellipsoid equation -1 ,SH,α parameter, can make many ellipsoids simulate the real person image in the image in space;

[0025] Step S4-2: speech deformation fitting training is performed when fitting the real-life image; the static image fitted by step S4-1 is the average image in space, and the connection between speech features and Gaussian ellipsoid model is established by training additional parameters, so that the trained Gaussian ellipsoid model can render accurate real-life action images; this step uses a multi-layer cross-attention mechanism, and generates the update amount of parameters such as the Hubert speech features and spatial position information extracted in step S2-3, and adds this update amount to the Gaussian ellipsoid set to obtain the real-life action modeling corresponding to the current audio.

[0026] Further, the step S5 specifically includes the following steps:

[0027] S5-1, generate audio; use the exclusive voice model obtained in step S3 to generate corresponding audio according to the input text;

[0028] S5-2, performing video frame rendering; using the audio generated in step S5-1 as input, using Hubert to extract input audio features, and using the 3D Gaussian Splatting model trained in step S4 to render a real-life dynamic video;

[0029] S5-3, post-processing; paste the character image of the real-life dynamic video back to the background data obtained in step S2-1, merge the video frames into a video, and merge the audio and video to obtain the final high-definition digital human video result.

[0030] The second object of the present invention is to provide a high-definition digital human video generation system for customizing a real-life image, comprising the following modules:

[0031] Collection module: used to collect audio and video data of the target person whose real-life image needs to be customized;

[0032] Preprocessing module: used to preprocess the audio and video data; the preprocessing includes segmenting the collected audio and video data to obtain the head data, torso data and background data of the portrait, then extracting the facial features including the blinking information, and finally extracting the audio data;

[0033] Customized voice model training module: used to customize a dedicated voice model based on audio data; first, the audio data in step S2 is segmented, and the segmented audio is recognized as annotated using a voice recognition model, and then trained to obtain a customized dedicated voice model;

[0034] A real person image fitting module is used to fit and generate a real person image for the preprocessed audio and video data; a 3D Gaussian Splatting model is trained through the head data, torso data, background data and facial features of the portrait preprocessed in step S2 to model a real person image, and a real person dynamic video is rendered through the 3D Gaussian Splatting model;

[0035] Video rendering module: used for combined rendering and output of audio and live-action video; input text, generate audio based on the exclusive voice model and input text, render live-action video at the same time, and then post-process and fuse the audio and live-action video and output them to obtain a high-definition digital human video with matching voice.

[0036] Furthermore, the acquisition module includes the following submodules:

[0037] Live video recording submodule: used to record live video; using a fixed camera position, record the audio and video data of the target person at the target perspective required for rendering; during the audio and video recording of the target person, make the target person speak according to the preset script, and record the spatial position information corresponding to the target spatial position when the target person pronounces different words;

[0038] Video editing submodule: used for video editing; cropping the recorded audio and video to center the area where the target person is located, as well as modifying the resolution and fixing the frame rate.

[0039] Furthermore, the preprocessing module specifically includes the following submodules:

[0040] Image region segmentation submodule: used to perform image region segmentation on audio and video data; separate the audio and video data obtained by the acquisition module into video frame images and audio data, and then use the face-parsing model to segment and obtain the face data, torso data and background data of each frame image;

[0041] Facial feature extraction submodule: used to extract facial features from video frame images; use OpenFace to extract AU45 encoding features containing blink information from each video frame image;

[0042] Audio feature extraction submodule: used to extract pronunciation features from audio data; the Hubert model is used to extract the pronunciation features of the target person in the audio.

[0043] Furthermore, the customized speech model training module specifically includes the following submodules:

[0044] Audio segmentation and annotation submodule: used to segment and annotate audio data; divide the audio into short audios of similar duration according to the time points of audio silence, and then use FunASR to recognize and annotate the segmented speech content;

[0045] Model training submodule: used to generate and train the speech model; use the data obtained from the audio segmentation and annotation submodule to fine-tune the pre-trained GPT-SoVITS model, complete the training, and obtain a dedicated speech model with customized timbre.

[0046] Furthermore, the real person image fitting module specifically includes the following submodules:

[0047] Static fitting training submodule: used to perform static fitting training when fitting the real person image, and fit the average static image of the customized image from the video image; this process uses multiple Gaussian ellipsoids to fit the static image, where each Gaussian ellipsoid is defined as follows:

[0048]

[0049] μ is the position vector of the ellipsoid space, ∑ -1 The covariance matrix contains the rotation and scale information of the ellipsoid in space; SH is the spherical harmonic function, which contains the color information presented when the ellipsoid is observed from different angles in space; α is the opacity of the ellipsoid;

[0050] Through image data training, adjust μ,∑ in the ellipsoid equation -1 ,SH,α parameter, can make many ellipsoids simulate the real person image in the image in space;

[0051] Speech deformation fitting training submodule: used for speech deformation fitting training when fitting and generating real-life images; the static image fitted by the static fitting training submodule is the average image in space, and by training additional parameters, the connection between speech features and Gaussian ellipsoid model is established, so that the trained Gaussian ellipsoid model can render accurate real-life action images; this step uses a multi-layer cross-attention mechanism, and generates the update amount of parameters such as the Hubert speech features and spatial position information extracted by the audio feature extraction submodule, and adds this update amount to the Gaussian ellipsoid set to obtain the real-life action modeling corresponding to the current audio.

[0052] Furthermore, the video rendering specifically includes the following submodules:

[0053] Text-to-speech submodule: used to generate audio; uses the exclusive speech model obtained by the custom speech model training module to generate corresponding audio based on the input text;

[0054] Video frame rendering submodule: used for video frame rendering; taking the audio generated by the text-to-speech submodule as input, using Hubert to extract input audio features, and using the 3D Gaussian Splatting model trained by the real-life image fitting module to render real-life dynamic video;

[0055] Post-processing submodule: used for post-processing; pasting the human figure of the real-life dynamic video back to the background data obtained by the image region segmentation submodule, merging the video frames into a video, and merging the audio and video to obtain the final high-definition digital human video result.

[0056] The third object of the present invention is to provide a computer-readable storage medium, which includes a program for generating a high-definition digital human video with a customized real-life image. When the program for generating a high-definition digital human video with a customized real-life image is executed by a processor, the steps of a method for generating a high-definition digital human video with a customized real-life image as described in any one of claims 1-6 are implemented.

[0057] A fourth object of the present invention is to provide a computer device, comprising the storage medium as claimed in claim 13, and a processor for executing a high-definition digital human video generation program of a customized real-life image in the storage medium.

[0058] Principles and advantages:

[0059] On the one hand, this solution refers to the GaussianTalker technology to achieve the reconstruction of real-life images and voice-driven rendering, and improves the use of the Chinese version of the Hubert model to extract voice features to achieve better Chinese voice-driven effects; on the other hand, this solution connects the complete digital human video production process in series, and adopts the most cutting-edge voice generation and digital human rendering methods to achieve low-cost, high-quality digital human video generation from scratch. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 The present invention is a flowchart of a method for generating a high-definition digital human video with a customized real-life image according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following is further described in detail through specific implementation methods:

[0062] Example

[0063] A method for generating a high-definition digital human video with a customized real-life image, basically as follows Figure 1 As shown, the following steps are included:

[0064] S1, collecting audio and video data of the target person whose real-life image needs to be customized; the step S1 includes the following steps:

[0065] S1-1, recording a live video; using a fixed camera position, recording the audio and video data of the target person at the target viewing angle required for rendering; during the audio and video recording of the target person, the target person speaks according to a preset script, and records the spatial position information corresponding to the target spatial position when the target person pronounces different words;

[0066] S1-2, perform video editing; crop the recorded audio and video so that the area where the target person is located is centered, and modify the resolution and fix the frame rate.

[0067] S2, preprocessing the audio and video data; the preprocessing includes segmenting the collected audio and video data to obtain the head data, torso data and background data of the portrait, then extracting the facial features including the blinking information, and finally extracting the audio data; the step S2 specifically includes the following steps:

[0068] S2-1, performing image region segmentation on the audio and video data; separating the audio and video data obtained in step S1 into video frame images and audio data, and then using the face-parsing model to segment and obtain face data, torso data and background data of each frame image;

[0069] S2-2, extract facial features from the video frame images; use OpenFace to extract the AU45 encoding features containing blink information from each video frame image;

[0070] S2-3. Extract pronunciation features from audio data; use the Hubert model to extract the pronunciation features of the target person in the audio.

[0071] S3, customizing a dedicated voice model according to the audio data; first, segmenting the audio data in step S2, and using the voice recognition model to recognize the segmented audio as annotations, and then training to obtain a customized dedicated voice model; the step S3 specifically includes the following steps:

[0072] S3-1: Segment and annotate the audio data; divide the audio into short audios of similar duration according to the silent time points of the audio, and then use FunASR to recognize and annotate the segmented speech content;

[0073] S3-2: Generate and train a voice model; use the data obtained in step S3-1 to fine-tune the pre-trained GPT-SoVITS model, complete the training, and obtain a dedicated voice model with customized timbre.

[0074] S4, fitting and generating a real person image for the pre-processed audio and video data; training a model based on 3D Gaussian Splatting through the head data, torso data, background data and facial features pre-processed in step S2, modeling a real person image, and rendering a real person dynamic video through the 3D Gaussian Splatting model; the step S4 specifically includes the following steps:

[0075] Step S4-1: Static fitting training is performed when fitting the real person image, and the average static image of the customized image is fitted from the video image; the process uses multiple Gaussian ellipsoids to fit the static image, wherein each Gaussian ellipsoid is defined as follows:

[0076] Formula 1:

[0077] μ is the position vector of the ellipsoid space, ∑ -1 The covariance matrix contains the rotation and scale information of the ellipsoid in space; SH is the spherical harmonic function, which contains the color information presented when the ellipsoid is observed from different angles in space; α is the opacity of the ellipsoid;

[0078] Through image data training, adjust μ,∑ in the ellipsoid equation -1 ,SH,α parameter, can make many ellipsoids simulate the real person image in the image in space;

[0079] Step S4-2: speech deformation fitting training is performed when fitting the real-life image; the static image fitted by step S4-1 is the average image in space, and the connection between speech features and Gaussian ellipsoid model is established by training additional parameters, so that the trained Gaussian ellipsoid model can render accurate real-life action images; this step uses a multi-layer cross-attention mechanism, and generates the update amount of parameters such as the Hubert speech features and spatial position information extracted in step S2-3, and adds this update amount to the Gaussian ellipsoid set to obtain the real-life action modeling corresponding to the current audio.

[0080] S5, performing combined rendering and output of audio and live action video; inputting text, generating audio based on the exclusive voice model and input text, rendering live action video at the same time, and then fusing the audio and live action video through post-processing and outputting them to obtain a high-definition digital human video matching the voice. The step S5 specifically includes the following steps:

[0081] S5-1, generate audio; use the exclusive voice model obtained in step S3 to generate corresponding audio according to the input text;

[0082] S5-2, performing video frame rendering; using the audio generated in step S5-1 as input, using Hubert to extract input audio features, and using the 3D Gaussian Splatting model trained in step S4 to render a real-life dynamic video;

[0083] S5-3, post-processing; paste the character image of the real-life dynamic video back to the background data obtained in step S2-1, merge the video frames into a video, and merge the audio and video to obtain the final high-definition digital human video result.

[0084] A high-definition digital human video generation system with customized real-life image, characterized by comprising the following modules:

[0085] Acquisition module: used to collect audio and video data of the target person whose real-life image needs to be customized; the acquisition module includes the following real-life video recording submodule and video editing submodule;

[0086] Live video recording submodule: used to record live video; using a fixed camera position, record the audio and video data of the target person at the target perspective required for rendering; during the audio and video recording of the target person, make the target person speak according to the preset script, and record the spatial position information corresponding to the target spatial position when the target person pronounces different words;

[0087] Video editing submodule: used for video editing; cropping the recorded audio and video to center the area where the target person is located, as well as modifying the resolution and fixing the frame rate.

[0088] Preprocessing module: used to preprocess the audio and video data; the preprocessing includes segmenting the collected audio and video data to obtain the head data, torso data and background data of the portrait, then extracting the facial features including the blinking information, and finally extracting the audio data; the preprocessing module specifically includes the following image region segmentation submodule, facial feature extraction submodule and audio feature extraction submodule;

[0089] Image region segmentation submodule: used to perform image region segmentation on audio and video data; separate the audio and video data obtained by the acquisition module into video frame images and audio data, and then use the face-parsing model to segment and obtain the face data, torso data and background data of each frame image;

[0090] Facial feature extraction submodule: used to extract facial features from video frame images; use OpenFace to extract AU45 encoding features containing blink information from each video frame image;

[0091] Audio feature extraction submodule: used to extract pronunciation features from audio data; the Hubert model is used to extract the pronunciation features of the target person in the audio.

[0092] Customized speech model training module: used to customize a dedicated speech model according to audio data; firstly, the audio data of step S2 is segmented, and the segmented audio is recognized as annotated using a speech recognition model, and then trained to obtain a customized dedicated speech model; the customized speech model training module specifically includes the following audio segmentation and annotation submodule and model training submodule;

[0093] Audio segmentation and annotation submodule: used to segment and annotate audio data; divide the audio into short audios of similar duration according to the time points of audio silence, and then use FunASR to recognize and annotate the segmented speech content;

[0094] Model training submodule: used to generate and train the speech model; use the data obtained from the audio segmentation and annotation submodule to fine-tune the pre-trained GPT-SoVITS model, complete the training, and obtain a dedicated speech model with customized timbre.

[0095] A real person image fitting module is used to fit and generate a real person image for the pre-processed audio and video data; a 3D Gaussian Splatting model is trained through the head data, torso data, background data and facial features of the portrait pre-processed in step S2 to model a real person image, and a real person dynamic video is obtained by rendering the 3D Gaussian Splatting model; the real person image fitting module specifically includes the following static fitting training submodule and speech deformation fitting training submodule;

[0096] Static fitting training submodule: used to perform static fitting training when fitting the real person image, and fit the average static image of the customized image from the video image; this process uses multiple Gaussian ellipsoids to fit the static image, where each Gaussian ellipsoid is defined as follows:

[0097]

[0098] μ is the position vector of the ellipsoid space, ∑ -1 The covariance matrix contains the rotation and scale information of the ellipsoid in space; SH is the spherical harmonic function, which contains the color information presented when the ellipsoid is observed from different angles in space; α is the opacity of the ellipsoid;

[0099] Through image data training, adjust μ,∑ in the ellipsoid equation -1 ,SH,α parameter, can make many ellipsoids simulate the real person image in the image in space;

[0100] Speech deformation fitting training submodule: used for speech deformation fitting training when fitting and generating real-life images; the static image fitted by the static fitting training submodule is the average image in space, and by training additional parameters, the connection between speech features and Gaussian ellipsoid model is established, so that the trained Gaussian ellipsoid model can render accurate real-life action images; this step uses a multi-layer cross-attention mechanism, and generates the update amount of parameters such as the Hubert speech features and spatial position information extracted by the audio feature extraction submodule, and adds this update amount to the Gaussian ellipsoid set to obtain the real-life action modeling corresponding to the current audio.

[0101] Video rendering module: used for combined rendering and output of audio and live action video; input text, generate audio based on the exclusive voice model and input text, render live action video at the same time, then fuse the audio and live action video through post-processing and output, and obtain a high-definition digital human video with matching voice. The video rendering specifically includes the following submodules: text-to-voice submodule, video frame rendering submodule, and post-processing submodule;

[0102] Text-to-speech submodule: used to generate audio; uses the exclusive speech model obtained by the custom speech model training module to generate corresponding audio based on the input text;

[0103] Video frame rendering submodule: used for video frame rendering; taking the audio generated by the text-to-speech submodule as input, using Hubert to extract input audio features, and using the 3D Gaussian Splatting model trained by the real-life image fitting module to render real-life dynamic video;

[0104] Post-processing submodule: used for post-processing; pasting the human figure of the real-life dynamic video back to the background data obtained by the image region segmentation submodule, merging the video frames into a video, and merging the audio and video to obtain the final high-definition digital human video result.

[0105] A computer-readable storage medium includes a program for generating a high-definition digital human video with a customized real-life image. When the program for generating a high-definition digital human video with a customized real-life image is executed by a processor, the steps of the method for generating a high-definition digital human video with a customized real-life image as described above are implemented.

[0106] Those skilled in the art can understand that all or part of the processes in the above-mentioned method for generating a high-definition digital human video with a customized real-life image can be completed by instructing related hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of various embodiments of the above-mentioned method for generating a high-definition digital human video with a customized real-life image. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0107] A computer device, comprising the storage medium and a processor, the processor being used to execute a program for generating a high-definition digital human video of a customized real-life image in the storage medium. The computer device can be implemented as various types of devices, such as a personal computer (PC), a server device, a mobile device, etc. The processor controls all functions of the electronic device by executing the program stored in the storage medium on the electronic device.

[0108] This solution addresses the difficulty of existing digital human technology in generating realistic high-definition videos of real-life images at low cost, as well as the problem that corresponding systems rarely reproduce and match real-life voices at low cost. The system connects and improves existing technologies, and achieves the goal of generating high-definition and realistic real-life digital human videos and cloning real-life voices with only a small amount of data.

[0109] The above is only an embodiment of the present invention. The common sense such as the known specific structure and characteristics in the scheme is not described in detail here. The ordinary technicians in the relevant field know all the common technical knowledge in the technical field of the invention before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent. The protection scope required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A method for generating a high-definition digital human video with a customized real-life image, characterized in that: The following steps are involved: S1. Collect audio and video data of the target person whose real-life image needs to be customized; S2, preprocessing the audio and video data; the preprocessing includes segmenting the collected audio and video data to obtain the head data, torso data and background data of the portrait, then extracting the facial features including the blinking information, and finally extracting the audio data; S3, customizing a dedicated voice model according to the audio data; first, segmenting the audio data in step S2, and using the voice recognition model to recognize the segmented audio as annotations, and then training to obtain a customized dedicated voice model; S4, fitting and generating a real person image for the preprocessed audio and video data; training a 3D Gaussian Splatting model with the head data, torso data, background data and facial features of the portrait preprocessed in step S2 to model a real person image, and rendering a real person dynamic video with the 3D Gaussian Splatting model; S5. Perform combined rendering and output of audio and real-person dynamic video; input text, generate audio based on the exclusive voice model and input text, render real-person dynamic video at the same time, and then fuse the audio and real-person dynamic video through post-processing and output to obtain a high-definition digital human video with matching voice.

2. The method for generating a high-definition digital human video with a customized real-life image according to claim 1, characterized in that: The step S1 comprises the following steps: S1-1, recording a live video; using a fixed camera position, recording the audio and video data of the target person at the target viewing angle required for rendering; during the audio and video recording of the target person, the target person speaks according to a preset script, and records the spatial position information corresponding to the target spatial position when the target person pronounces different words; S1-2, perform video editing; crop the recorded audio and video so that the area where the target person is located is centered, and modify the resolution and fix the frame rate.

3. The method for generating a high-definition digital human video with a customized real-life image according to claim 2, characterized in that: The step S2 specifically includes the following steps: S2-1, performing image region segmentation on the audio and video data; separating the audio and video data obtained in step S1 into video frame images and audio data, and then using the face-parsing model to segment and obtain face data, torso data and background data of each frame image; S2-2, extract facial features from the video frame images; use OpenFace to extract the AU45 encoding features containing blink information from each video frame image; S2-3. Extract pronunciation features from audio data; use the Hubert model to extract the pronunciation features of the target person in the audio.

4. The method for generating a high-definition digital human video with a customized real-life image according to claim 3, characterized in that: The step S3 specifically comprises the following steps: S3-1: Segment and annotate the audio data; divide the audio into short audios of similar duration according to the silent time points of the audio, and then use FunASR to recognize and annotate the segmented speech content; S3-2: Generate and train a voice model; use the data obtained in step S3-1 to fine-tune the pre-trained GPT-SoVITS model, complete the training, and obtain a dedicated voice model with customized timbre.

5. The method for generating a high-definition digital human video with a customized real-life image according to claim 4, characterized in that: The step S4 specifically comprises the following steps: Step S4-1: Static fitting training is performed when fitting the real person image, and the average static image of the customized image is fitted from the video image; the process uses multiple Gaussian ellipsoids to fit the static image, wherein each Gaussian ellipsoid is defined as follows: μ is the position vector of the ellipsoid space, ∑ -1 The covariance matrix contains the rotation and scale information of the ellipsoid in space; SH is the spherical harmonic function, which contains the color information presented when the ellipsoid is observed from different angles in space; α is the opacity of the ellipsoid; Through image data training, adjust μ,∑ in the ellipsoid equation -1 ,SH,α parameter, can make many ellipsoids simulate the real person image in the image in space; Step S4-2: speech deformation fitting training is performed when fitting the real-life image; the static image fitted by step S4-1 is the average image in space, and the connection between speech features and Gaussian ellipsoid model is established by training additional parameters, so that the trained Gaussian ellipsoid model can render accurate real-life action images; this step uses a multi-layer cross-attention mechanism, and generates the update amount of parameters such as the Hubert speech features and spatial position information extracted in step S2-3, and adds this update amount to the Gaussian ellipsoid set to obtain the real-life action modeling corresponding to the current audio.

6. The method for generating a high-definition digital human video with a customized real-life image according to claim 5, characterized in that: The step S5 specifically comprises the following steps: S5-1, generate audio; use the exclusive voice model obtained in step S3 to generate corresponding audio according to the input text; S5-2, performing video frame rendering; using the audio generated in step S5-1 as input, using Hubert to extract input audio features, and using the 3D Gaussian Splatting model trained in step S4 to render a real-life dynamic video; S5-3, post-processing; paste the character image of the real-life dynamic video back to the background data obtained in step S2-1, merge the video frames into a video, and merge the audio and video to obtain the final high-definition digital human video result.

7. A high-definition digital human video generation system for customizing real-life images, characterized in that: Includes the following modules: Collection module: used to collect audio and video data of the target person whose real-life image needs to be customized; Preprocessing module: used to preprocess the audio and video data; the preprocessing includes segmenting the collected audio and video data to obtain the head data, torso data and background data of the portrait, then extracting the facial features including the blinking information, and finally extracting the audio data; Customized voice model training module: used to customize a dedicated voice model based on audio data; first, the audio data in step S2 is segmented, and the segmented audio is recognized as annotated using a voice recognition model, and then trained to obtain a customized dedicated voice model; A real person image fitting module is used to fit and generate a real person image for the preprocessed audio and video data; a 3D Gaussian Splatting model is trained through the head data, torso data, background data and facial features of the portrait preprocessed in step S2 to model a real person image, and a real person dynamic video is rendered through the 3D Gaussian Splatting model; Video rendering module: used for combined rendering and output of audio and live-action video; input text, generate audio based on the exclusive voice model and input text, render live-action video at the same time, and then post-process and fuse the audio and live-action video and output them to obtain a high-definition digital human video with matching voice.

8. The high-definition digital human video generation system for customizing a real-life image according to claim 7, characterized in that: The acquisition module Includes the following submodules: Live video recording submodule: used to record live video; Using a fixed camera position, record the audio and video data of the target person at the target viewing angle required for rendering; during the audio and video recording of the target person, make the target person speak according to the preset script, and record the spatial position information corresponding to the target spatial position when the target person pronounces different words; Video editing submodule: used for video editing; cropping the recorded audio and video to center the area where the target person is located, as well as modifying the resolution and fixing the frame rate.

9. The high-definition digital human video generation system for customizing a real-life image according to claim 8, characterized in that: The preprocessing module specifically Includes the following submodules: Image region segmentation submodule: used to perform image region segmentation on audio and video data; separate the audio and video data obtained by the acquisition module into video frame images and audio data, and then use the face-parsing model to segment and obtain the face data, torso data and background data of each frame image; Facial feature extraction submodule: used to extract facial features from video frame images; Use OpenFace to extract the AU45 encoding features containing blink information from each video frame image; Audio feature extraction submodule: used to extract pronunciation features from audio data; Use the Hubert model to extract the pronunciation features of the target person in the audio.

10. The high-definition digital human video generation system for customizing a real-life image according to claim 9, characterized in that: The customized speech model training module specifically includes the following submodules: Audio segmentation and annotation submodule: used to segment and annotate audio data; divide the audio into short audios of similar duration according to the time points of audio silence, and then use FunASR to recognize and annotate the segmented speech content; Model training submodule: used to generate and train the speech model; use the data obtained from the audio segmentation and annotation submodule to fine-tune the pre-trained GPT-SoVITS model, complete the training, and obtain a dedicated speech model with customized timbre.

11. The system for generating high-definition digital human videos with customized real-life images according to claim 10, characterized in that: The real person image fitting module specifically Includes the following submodules: Static fitting training submodule: used to perform static fitting training when fitting the real person image, and fit the average static image of the customized image from the video image; this process uses multiple Gaussian ellipsoids to fit the static image, where each Gaussian ellipsoid is defined as follows: μ is the position vector of the ellipsoid space, ∑ -1 The covariance matrix contains the rotation and scale information of the ellipsoid in space; SH is the spherical harmonic function, which contains the color information presented when the ellipsoid is observed from different angles in space; α is the opacity of the ellipsoid; Through image data training, adjust μ,∑ in the ellipsoid equation -1 ,SH,α parameter, can make many ellipsoids simulate the real person image in the image in space; Speech deformation fitting training submodule: used for speech deformation fitting training when fitting and generating real-life images; The static image fitted by the static fitting training submodule is the average image in space. By training additional parameters, the connection between speech features and Gaussian ellipsoid model is established, so that the trained Gaussian ellipsoid model can render accurate real-life action images. This step uses a multi-layer cross-attention mechanism, and generates the update amount of parameters such as the Hubert speech features extracted by the audio feature extraction submodule and the spatial position information, and adds this update amount to the Gaussian ellipsoid set to obtain the real-life action modeling corresponding to the current audio.

12. The high-definition digital human video generation system for customizing a real-life image according to claim 11, characterized in that: The video rendering specifically includes the following submodules: Text-to-speech submodule: used to generate audio; Use the exclusive voice model obtained by the custom voice model training module to generate corresponding audio based on the input text; Video frame rendering submodule: used for video frame rendering; taking the audio generated by the text-to-speech submodule as input, using Hubert to extract input audio features, and using the 3D Gaussian Splatting model trained by the real-life image fitting module to render real-life dynamic video; Post-processing submodule: used for post-processing; pasting the human figure of the real-life dynamic video back to the background data obtained by the image region segmentation submodule, merging the video frames into a video, and merging the audio and video to obtain the final high-definition digital human video result.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a customized high-definition digital human video generation program for a real-life image. When the customized high-definition digital human video generation program for a real-life image is executed by a processor, the steps of a customized high-definition digital human video generation method for a real-life image as described in any one of claims 1-6 are implemented.

14. A computer device, characterized in that: The computer device includes the storage medium as claimed in claim 13, and a processor, wherein the processor is used to execute a high-definition digital human video generation program of a customized real-life image in the storage medium.

Images