Digital human video generation method and device, storage medium and program product

Abstract

The embodiment of the invention provides a digital human video generation method and device, a storage medium and a program product. In the embodiment of the invention, the original reasoning environment of the target digital human reasoning model is changed, reasoning operation at the server side is changed into reasoning at the mobile terminal, and the reasoning speed of the target digital human model is improved. Besides, a plurality of target face area images with continuously changed mouth forms are obtained through reasoning based on a plurality of audio feature segments corresponding to the audio data by using a target digital human reasoning model, and a digital human action picture is generated according to the plurality of target digital human images with continuously changed mouth forms. And driving the digital person in the digital person action picture to play the audio data to obtain a target digital person video, thereby not only improving the adaptation degree of the mouth shape and the audio content, but also reducing the delay and lagging problems of playing the digital person video, and improving the playing quality of the target digital person video.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a method, device, storage medium and program product for generating digital human videos. Background Art

[0002] With the continuous development of artificial intelligence, AI digital human technology has also been widely used. At present, most solutions for generating digital humans rely on the server to complete the reasoning task and then transmit the generated AI digital human video to the mobile terminal.

[0003] However, this approach has the following problems: when the server needs to perform multiple reasoning tasks at the same time, the server may not respond in time due to heavy load, resulting in slow reasoning speed. In addition, after the server generates the digital human video, it needs to be transmitted to the mobile terminal through the network. If the network environment is poor, it may cause delays and freezes in playing the digital human video on the mobile terminal. Summary of the invention

[0004] Multiple aspects of the present application provide a digital human video generation method, device, storage medium and program product to improve the model reasoning speed and improve the playback quality of digital human videos.

[0005] The embodiment of the present application provides a method for generating a digital human video, which is applied to a mobile terminal, and the mobile terminal runs a target digital human reasoning model; the method comprises: obtaining audio data for generating a digital human voice, an image resource package for generating a digital human image, and a frame rate of a digital human video to be generated, wherein the image resource package comprises a plurality of original digital human base maps, an original face region image in each digital human base map, and position information of each original face region image in the corresponding original digital human base map; converting the audio data into target audio feature information required by the target digital human reasoning model, and performing a segmentation operation on the target audio feature information according to the duration and frame rate of the audio data to obtain a plurality of audio feature information; The method comprises the steps of: inputting a plurality of audio feature segments into a target digital human inference model, mapping the plurality of audio feature segments into a plurality of mouth deformation features, and generating a plurality of target face region images with continuously changing mouth shapes based on the plurality of mouth deformation features; replacing the original face region images in the plurality of original digital human base images with the plurality of target face region images according to the position information of each original face region image in the corresponding original digital human base image, so as to obtain a plurality of target digital human images with continuously changing mouth shapes; generating a digital human action picture according to the plurality of target digital human images with continuously changing mouth shapes, and driving the digital human mouth in the digital human action picture to broadcast audio data, so as to obtain the target digital human video body;

[0006] The application embodiment also provides an electronic device, including: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the above-mentioned digital human live broadcast method.

[0007] The embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements the steps in the digital human live broadcast method.

[0008] The embodiment of the present application also provides a computer program product, which includes a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is enabled to implement the steps in the above-mentioned digital human live broadcast method.

[0009] In the embodiment of the present application, the original reasoning environment of the target digital human reasoning model is changed from reasoning operation on the server side to reasoning on the mobile terminal, which can avoid the problem that when the server side needs to perform multiple reasoning tasks at the same time, the server side may not respond in time due to excessive load, resulting in slow reasoning speed, thereby improving the reasoning speed of the target digital human model. In addition, the target digital human reasoning model is used to infer multiple target facial area images with continuously changing mouth shapes based on multiple audio feature fragments corresponding to audio data, and a digital human action picture is generated based on multiple target digital human images with continuously changing mouth shapes, and the digital human in the digital human action picture is driven to broadcast audio data to obtain the target digital human video, which can not only improve the adaptability of the mouth shape and the audio content, but also reduce the delay and jamming problems of playing the digital human video, and improve the playback quality of the target digital human video. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0011] Figure 1 A flowchart of a method for generating a digital human video provided by an exemplary embodiment of the present application;

[0012] Figure 2 A flowchart of a model conversion process provided by an exemplary embodiment of the present application;

[0013] Figure 3 A schematic structural diagram of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0015] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0016] In view of the existing technical problems such as slow reasoning speed of the server side and delay and jamming of playing digital human videos on mobile terminals, in the embodiment of the present application, the original reasoning environment of the target digital human reasoning model is changed from reasoning operation on the server side to reasoning on the mobile terminal, which can avoid the problem that when the server side needs to perform multiple reasoning tasks at the same time, the server side may not respond in time due to excessive load, resulting in slow reasoning speed, thereby improving the reasoning speed of the target digital human model. In addition, the target digital human reasoning model is used to infer multiple target facial area images with continuously changing mouth shapes based on multiple audio feature fragments corresponding to audio data, and a digital human action picture is generated based on multiple target digital human images with continuously changing mouth shapes, and the digital human in the digital human action picture is driven to broadcast audio data to obtain the target digital human video, which can not only improve the adaptability of the mouth shape and the audio content, but also reduce the delay and jamming problems of playing the digital human video, and improve the playback quality of the target digital human video.

[0017] A solution provided by an embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0018] Figure 1 The following is a flow chart of a method for generating a digital human video provided by an exemplary embodiment of the present application. Figure 1 As shown, the method is applied to a mobile terminal, and the mobile terminal runs a digital human reasoning model. For the convenience of description and distinction, the digital human reasoning model is referred to as a target digital human reasoning model. The method includes:

[0019] 101. Acquire audio data for generating digital human voice, an image resource package for generating digital human image, and a frame rate of a digital human video to be generated, wherein the image resource package includes a plurality of original digital human base maps, an original face region image in each digital human base map, and position information of each original face region image in the corresponding original digital human base map;

[0020] 102. Convert the audio data into target audio feature information required by the target digital human reasoning model, and segment the target audio feature information according to the duration and frame rate of the audio data to obtain multiple audio feature segments;

[0021] 103. Inputting a plurality of audio feature segments into a target digital human inference model, mapping the plurality of audio feature segments into a plurality of mouth deformation features, and generating a plurality of target facial region images with continuously changing mouth shapes based on the plurality of mouth deformation features;

[0022] 104. According to the position information of each original face region image in the corresponding original digital human base image, the original face region images in the multiple original digital human base images are replaced with the multiple target face region images to obtain multiple target digital human images with continuously changing mouth shapes;

[0023] 105. Generate a digital human action picture based on a plurality of target digital human images with continuously changing mouth shapes, and drive the digital human mouth in the digital human action picture to broadcast audio data to obtain a target digital human video.

[0024] In the embodiments of the present application, there is no limitation on the implementation form of the mobile terminal. The mobile terminal may be a smart handheld device, such as a smart phone, a tablet computer, a laptop computer or a desktop computer, etc.; or, the mobile terminal may also be a smart wearable device, such as a smart watch, a smart bracelet, etc.; or, the mobile terminal may also be various smart home appliances with display screens, such as a smart TV, a smart large screen or a smart robot, etc.

[0025] In an embodiment of the present application, audio data refers to audio data of voice content contained in the audio, which may be recorded in advance, or may be audio data generated in advance or in real time by a server or mobile terminal based on text information. The timing of generating audio data depends on the specific application scenario, and this embodiment does not specifically limit this. For example, the real-time and flexible conversation audio data in an AI interview room may be audio data for answering generated in real time based on text information corresponding to the input content of the dialogue party, and the audio data corresponding to the fixed words in some dialogue scenarios may be recorded in advance or generated in advance based on text information.

[0026] In order to determine the deformation of the mouth with audio data, this embodiment introduces a digital human reasoning model, which is at least an AI model for inferring the mouth shape based on audio data. Specifically, it can predict and generate mouth shape animation synchronized with the audio data based on the input audio data, such as being able to generate realistic animations of digital humans or virtual characters, so that the characters can move their mouths naturally according to the audio data, thereby improving the accuracy of identifying mouth deformation. The embodiment of this application does not limit the type of digital human reasoning model. For example, the digital human reasoning model can be Wav2Lip, ER-NeRF, VideoRetalking, MuseTalk, EchoMimic, and so on. Among them, Wav2Lip is an open source lip synchronization tool that can generate lip animations that match the input speech. It is suitable for any face and any language, and can generate realistic lip movements; ER-NeRF is a real-time responsive Talking Portrait Synthesis model that can synchronize the given audio with the mouth shape of the character in the video. It should be noted that specific videos need to be trained to generate the corresponding digital human reasoning model; VideoRetalking can synchronize the mouth shape of the video character with the input sound. The model includes face video generation, audio-driven lip synchronization and face enhancement; MuseTalk is a real-time audio-driven lip synchronization model that supports multiple language inputs and is particularly suitable for high-definition facial areas; EchoMimic can give static images vivid voices and expressions. It combines audio and facial landmarks through deep learning models to create highly realistic dynamic portrait videos. These models generate corresponding lip animations by analyzing the input voice signals, so that the mouth shape of the digital human character when speaking matches the actual voice content. While improving the accuracy of recognizing mouth deformation, it can also improve the realism of the character and the immersion of the audience. For the convenience of distinction and description, the digital human reasoning model in actual application is referred to as the target digital human reasoning model. In this embodiment, the focus is on the Wav2Lip model.

[0027] It should be noted that the target digital human reasoning model is obtained by converting the model format of the initial digital human reasoning model, and the initial digital human reasoning model is developed on the server side through a target model development framework suitable for the server side. Specifically, when the mobile terminal has a demand for the target digital human model, a model conversion request message is sent to the server side, and the model conversion request message includes the system type of the mobile terminal, available resource information, and model functions required by the mobile terminal, so as to request the server side to convert the initial digital human reasoning model into the target digital human reasoning model based on the system type, available resource information, and model functions; further, a model conversion response message is received from the server side, and the model conversion response message is used to notify the mobile terminal to obtain the model file of the target digital human reasoning model from the remote persistent storage space; the model file of the target digital human reasoning model is obtained from the remote persistent storage space and stored in the local persistent storage space, and when the target digital human reasoning model is used, the model file is loaded from the local persistent storage space to run the target digital human reasoning model.

[0028] It should also be noted that the development process of the initial digital human model may include the training process of the initial digital human model, or the development process of the initial digital human model may not include the training process of the initial digital human model. In the case where the development process of the initial digital human model does not include the training process of the initial digital human model, the training process of the initial digital human model may be completed on the server side before the model format conversion, or on the mobile terminal before the model format conversion, or on the mobile terminal after the model format conversion. The development, training and model format conversion process of the initial digital human reasoning model can be referred to the relevant description of the following embodiments, which will not be repeated here.

[0029] In the embodiment of the present application, the specific implementation method of the target digital human reasoning model running on the mobile terminal is not limited. For example, the mobile terminal can directly run the target digital human reasoning model; or, the target digital human reasoning model runs on an application on the mobile terminal. For the convenience of description and distinction, the application is referred to as the target application. In addition, this embodiment does not limit the implementation form of the target application. The target application can be an independently running APP, or a web page or a small program that depends on the APP to run.

[0030] In an embodiment of the present application, in order to obtain a digital human video, audio data for generating the digital human voice and an image resource package for generating the digital human image may be first obtained, so as to infer the mouth shape changes corresponding to the digital human image based on the audio data, and further obtain the digital human video based on the audio data and the mouth shape changes. Among them, the image resource package may only contain multiple original digital human base maps, or may simultaneously contain multiple original digital human base maps, original facial area images in each digital human base map, and position information of each original facial area image in the corresponding original digital human base map, which is not limited in this embodiment. Multiple original digital human base maps may include multiple groups of original digital human base maps, and each group of original digital human base maps may be multiple continuous digital human images captured from any digital human video, and the mouth shapes in the multiple continuous digital human images are continuously changing.

[0031] The original data format of the audio data may be different from the data format required by the target digital human reasoning model. After obtaining the audio data, in order to facilitate the target digital human reasoning model to infer the mouth shape changes corresponding to the audio data, the original data format of the audio data can be converted to the data format required by the target digital human reasoning model, that is, the initial audio feature information of the audio data is converted to the target audio feature information required by the target digital human reasoning model. It should be noted that the types of audio feature information required by different digital human reasoning models may be the same or different. Taking the Wav2Lip model as an example, the original data format of the audio data is .wav format, and the data format required by the Wav2Lip model is the Mel spectrum data format. Then, the audio data needs to be converted from .wav format to Mel spectrum data format, that is, the initial audio feature information of the audio data is converted to the Mel spectrum data required by the Wav2Lip model. The Mel spectrum data corresponds to a Mel spectrum graph.

[0032] In an embodiment of the present application, audio data is converted into audio feature information required by a target digital human reasoning model, including: extracting Fourier transform data from the audio data; generating Mel spectrum data based on the Fourier transform data; and cropping, padding or resizing the Mel spectrum data according to the requirements of the target digital human reasoning model for input data to obtain target audio feature information. Among them, cropping refers to removing edge noise in the Mel spectrum data or retaining the required spectrum portion. Padding refers to adding zero values ​​or other specific values ​​to the edges of the data. Usually, when the size of the Mel spectrum data does not meet the input requirements of the model, the size of the data can be adjusted by padding to meet the requirements. Sizing refers to resampling or interpolation to change the resolution or size of the Mel spectrum data to adapt it to a specific input format. The target audio feature information refers to the Mel spectrum feature information corresponding to the Mel spectrum data processed based on the requirements of the input data of the target digital human reasoning model.

[0033] In an optional embodiment, Fourier transform data is extracted from audio data, and spectrum data is generated according to the Fourier transform data, including: first, the audio data needs to be loaded using an appropriate library, such as using the Librosa library to load audio data in .wav format; preprocessing the audio data, the preprocessing includes but is not limited to normalizing the volume, removing noise and other steps to improve the accuracy of subsequent processing; converting the audio data into an audio signal, the audio signal corresponding to key feature information; performing a short-time Fourier transform (Fast Fourier Transform, FFT) on the audio signal to convert the audio signal from the time domain to the frequency domain; using a Mel filter to convert the audio signal in the frequency domain to the Mel scale to generate a Mel spectrum map, the Mel spectrum map corresponding to the Mel spectrum data. Further, the Mel spectrum map can also be subjected to a discrete cosine transform (Discrete Cosine Transform, DCT) in English, which is the full name of extracting Mel frequency cepstral coefficients (Mel Frequency Cepstral Coefficients, MFCCs), the Mel frequency cepstral coefficient is a feature representation of the Mel spectrum data, that is, the Mel frequency cepstral coefficient is the Mel spectrum feature information corresponding to the Mel spectrum data.

[0034] Further optionally, in order to improve the accuracy of the target digital human reasoning model's reasoning, the target audio feature information can be segmented, and the target digital human reasoning model can be used to reason about each segment of audio feature information separately to obtain multiple digital human images with continuously changing mouth shapes.

[0035] In an optional embodiment, the target feature information can be segmented according to the duration of the audio data and the frame rate of the digital human video to be generated. The frame rate refers to the number of static images (frames) that can be displayed per second during video playback. The frame rate is an important indicator of the smoothness of video playback, which is usually measured in "frames per second" (FPS). For example, if the frame rate of the digital human to be generated is 25fps, 25 pictures will be played and explained within 1 second. Based on this, the target audio feature information is segmented, including: obtaining the frame rate of the digital human video to be generated; according to the duration of the audio data and the frame rate of the digital human video to be generated, the target feature information is segmented to obtain multiple audio feature fragments.

[0036] Among them, when obtaining the frame rate of the digital human video to be generated, it is necessary to consider the purpose of the digital human video, the audience's viewing device, the video playback platform, and the required visual effects. For example, if the digital human video is used for online playback and needs to show smooth facial expressions and synchronized lip movements, a frame rate of 30fps or higher can be selected; if the digital human video is used in a movie or art project, 24fps can be selected to meet the viewing experience. In addition, technical limitations and costs need to be considered. Usually, a higher frame rate means that more data needs to be processed during editing and rendering, which may increase the demand and cost of computing resources. At the same time, the size of the high-frame-rate digital human video file will also increase, which may affect storage and transmission efficiency. In practical applications, if applied to cloud intelligent digital human services, the video frame rate is generally not less than 25fps and not higher than 60fps. If applied to silicon-based digital human image cloning, the video frame rate is generally 30fps or 60fps, so that the smoothness and detail of the video can be ensured while taking into account the technical and cost limitations in practical applications.

[0037] Optionally, the target audio feature information is segmented according to the duration and frame rate of the audio data to obtain multiple audio feature segments, including: taking the duration of the audio data as the dividend and the frame rate as the divisor, performing a quotient operation to obtain the video frame duration, which means the duration of each frame of the video; taking the video frame duration as the segmentation unit, the target audio feature information is segmented to obtain multiple audio feature segments. For example, if the duration of the audio data is 50s and the frame rate of the digital human video to be generated is 25fps, then

[0038] Furthermore, after obtaining multiple audio feature fragments, the multiple audio feature fragments can be input into the target digital human inference model, the multiple audio feature fragments can be mapped into multiple mouth deformation features, and multiple target facial area images with continuously changing mouth shapes can be generated based on the multiple mouth deformation features.

[0039] In an embodiment of the present application, multiple audio feature fragments are input into a target digital human inference model, the multiple audio feature fragments are mapped into multiple mouth deformation features, and multiple target facial area images with continuously changing mouth shapes are generated based on the multiple mouth deformation features, including: inputting multiple audio feature fragments into a target digital human inference model, inferring the changes in the digital human's mouth shape based on the multiple audio feature fragments, and obtaining multiple mouth deformation features; converting the multiple mouth deformation features into multiple pixel point data in the facial area and position distribution information of each pixel point; generating multiple target facial area images according to the multiple pixel point data in the facial area corresponding to the multiple mouth deformation features and the position distribution information of each pixel point.

[0040] In an optional embodiment, the target digital human inference model includes: an audio feature processing network layer, which infers the changes in the digital human's mouth shape based on multiple audio feature segments to obtain multiple mouth deformation features, including: inputting multiple audio feature segments into the audio feature processing network layer to generate mouth deformation features corresponding to each audio feature segment. Among them, each mouth deformation feature is a multidimensional feature vector, and a multidimensional feature vector at least includes a mouth shape parameter, an opening and closing degree parameter, and a lip position parameter. The mouth shape parameter is used to represent the current mouth shape, the opening and closing degree parameter is used to represent the current mouth opening and closing degree, and the lip position parameter is used to represent the current mouth position in the facial area.

[0041] Further optionally, the target digital human inference model may also include: a visual feature processing network layer, which converts multiple mouth deformation features into multiple pixel point data in the facial area and position distribution information of each pixel point, including: inputting multiple mouth deformation features into the video feature processing network layer, and based on the mouth shape parameters, opening and closing parameters and lip position parameters corresponding to each mouth deformation feature, converting each mouth deformation feature into multiple pixel point data in the facial area and position distribution information of each pixel point.

[0042] Further optionally, the target digital human inference model also includes a generator. After obtaining multiple pixel point data and position distribution information of each pixel point, the multiple pixel point data and the position distribution information of each pixel point can be input into the generator, and based on the multiple pixel point data and the position distribution information of each pixel point, multiple target facial area images with continuously changing mouth shapes can be obtained.

[0043] It can be seen from the above embodiments that the image resource package includes multiple original digital human base maps, original facial area images in each digital human base map, and position information of each original facial area image in the corresponding original digital human base map. After obtaining multiple target facial area images with continuously changing mouth shapes, the original facial area images in the multiple original digital human base maps can be replaced with the multiple target facial area images according to the position information of each original facial area image in the corresponding original digital human base map to obtain multiple target digital human images with continuously changing mouth shapes.

[0044] Furthermore, after obtaining a plurality of target digital human images with continuously changing mouth shapes, a target digital human video can be obtained based on the plurality of target digital human images with continuously changing mouth shapes. This embodiment does not limit the specific implementation method of obtaining the target digital human video based on the plurality of target digital human images with continuously changing mouth shapes, and different implementation methods may correspond to different application scenarios.

[0045] In an optional embodiment, a target digital human video is obtained based on a plurality of target digital human images with continuously changing mouth shapes, including: generating a digital human action picture based on the plurality of target digital human images with continuously changing mouth shapes, and driving the digital human mouth in the digital human action picture to broadcast the audio data to obtain the target digital human video. The digital human action picture and the broadcast audio in the target digital human video are two independent parts, which is suitable for real-time dialogue scenarios, such as real-time question and answer in AI live broadcast rooms / interview rooms.

[0046] In another optional embodiment, a target digital human video is obtained according to a plurality of target digital human images with continuously changing mouth shapes, including: generating a target digital human video according to the plurality of target digital human images with continuously changing mouth shapes and audio data for generating digital human voices. The target digital human video is a unified whole with the action picture of the digital human and the spoken audio, which is suitable for non-real-time playback scenarios, such as non-real-time live broadcast digital human live video. In this embodiment, correspondingly, the generator in the target digital human reasoning model can also obtain the target digital human video according to the plurality of target digital human images with continuously changing mouth shapes. Further, the target digital human reasoning model can also include a lip synchronization discriminator for evaluating the synchronization accuracy between the generated video frame and the audio data. In the case where the evaluation result does not meet the synchronization accuracy threshold, the target digital human reasoning model can be retrained based on the synchronization accuracy standard, and the target digital human video is generated using the retrained target digital human reasoning model based on the plurality of target digital human images with continuously changing mouth shapes and the audio data for generating digital human voices. Furthermore, the target digital human reasoning model may also include a visual quality discriminator for evaluating whether the generated video frames look natural visually and have no obvious generation traces. Similarly, if the evaluation result does not meet the synchronization accuracy threshold, the target digital human reasoning model can be retrained based on the synchronization accuracy standard, and the retrained target digital human reasoning model can be used to generate the target digital human video based on multiple target digital human images with continuously changing mouth shapes and audio data for generating digital human voices.

[0047] In the above embodiments of the present application, the original reasoning environment of the target digital human reasoning model is changed from reasoning operation on the server side to reasoning on the mobile terminal, which can avoid the problem that when the server side needs to perform multiple reasoning tasks at the same time, the server side may not respond in time due to excessive load, resulting in slow reasoning speed, thereby improving the reasoning speed of the target digital human model. In addition, the target digital human reasoning model is used to infer multiple target facial area images with continuously changing mouth shapes based on multiple audio feature segments corresponding to audio data, and a digital human action picture is generated based on multiple target digital human images with continuously changing mouth shapes, and the digital human in the digital human action picture is driven to broadcast audio data to obtain the target digital human video, which can not only improve the adaptability of the mouth shape and the audio content, but also reduce the delay and jamming problems of playing the digital human video, thereby improving the playback quality of the target digital human video.

[0048] Combine the following Figure 2 The conversion process of the target digital human model is explained in detail.

[0049] S1. receiving a model conversion request message sent by a mobile terminal, where the model conversion request message includes at least a system type of the mobile terminal, available resource information, and a model function required by the mobile terminal;

[0050] S2. determining an initial digital human reasoning model adapted to the system type and required model functions of the mobile terminal from the digital human reasoning model developed based on the target model development framework, wherein the target model development framework refers to a model development framework applicable to the server;

[0051] S3. When the available resource information of the mobile terminal meets the resource loading conditions required by the data packet, determine the target format conversion tool according to the original model format supported by the initial digital human reasoning model and the model-format conversion tool mapping table, wherein the model-format conversion tool mapping table stores format conversion tools corresponding to digital human reasoning models supporting different model formats;

[0052] S4. Calling a target format conversion tool to convert the initial digital human reasoning model into a model format, so as to obtain a target digital human reasoning model that supports the target model format, and the target digital human reasoning model is suitable for running on a mobile terminal.

[0053] In the embodiment of the present application, the system type includes but is not limited to: Android system, IOS system or Hongmeng system. Available resource information includes but is not limited to: CPU information, such as model and speed, number of cores and threads, and cache size; memory (RAM) information, such as total memory capacity, used and available memory, memory speed, and memory timing; storage information, such as hard disk or solid state drive capacity, available space, read and write speed, and interface type (such as SATA, NVMe); graphics processing unit (GPU) information, such as model, video memory size, and GPU usage; network information, such as network adapter type, IP address, network bandwidth, and network connection status; hardware status, such as device health status and fault information.

[0054] In an embodiment of the present application, an initial digital human reasoning model adapted to the system type and required model functions of a mobile terminal is determined from digital human reasoning models developed based on a target model development framework, including: selecting a first candidate digital human reasoning model adapted to the system type of the mobile terminal from digital human reasoning models developed based on a target model development framework, and different digital human reasoning models are configured with a list of operating systems adapted thereto; determining a second candidate digital human reasoning model adapted to the model functions required by the mobile terminal based on the functional description information of the first candidate digital human reasoning model; if there is one second candidate digital human reasoning model, using the second candidate digital human reasoning model as the initial digital human reasoning model; if there are multiple second candidate digital human reasoning models, selecting the one with the smallest model file as the initial digital human reasoning model.

[0055] Further optionally, a model file of the initial digital human reasoning model is obtained; based on the size of the model file, it is determined whether the bandwidth information and / or memory information in the available resource information of the mobile terminal meets the minimum bandwidth value and / or minimum memory amount required in the resource loading condition; if so, it is determined that the available resource information of the mobile terminal meets the resource loading condition.

[0056] In an embodiment of the present application, a target format conversion tool is called to convert the model format of the initial digital human reasoning model to obtain a target digital human reasoning model that supports the target model format, including: calling the target format conversion tool to parse the model file of the initial digital human reasoning model to obtain the network structure and weight parameter set of the initial digital human reasoning model, the network structure is used to describe the multiple network layers included in the initial digital human reasoning model and the connection relationship between the multiple network layers, the weight parameter set includes the original weight parameters of each network layer, and the weight parameters are used to indicate the degree of influence of the input data on the model output. Specifically, the weight determines the degree to which the input data should be amplified or reduced before being passed to the next layer; and, by simulating the initial digital human reasoning model or performing control flow analysis, a static calculation graph of the initial digital human reasoning model is obtained, the static calculation graph includes the processing logic of each network layer from input data to output data and the data flow dependency relationship between network layers; according to the target model format, the network structure, weight parameter set and static calculation graph of the initial digital human reasoning model are reorganized to obtain the target digital human reasoning model.

[0057] In an embodiment of the present application, the model file of the initial digital human reasoning model is parsed to obtain the network structure and weight parameter set of the initial digital human reasoning model, including: parsing a sub-file with a specific format in the model file to obtain the weight parameter set of the initial digital human reasoning model; defining the model structure of the initial digital human reasoning model based on the forward propagation function in the model file to obtain the network structure of the initial digital human reasoning model.

[0058] In an embodiment of the present application, a static computational graph of the initial digital human reasoning model is obtained by simulating the operation of the initial digital human reasoning model, including: inputting preset exemplary input data into the initial digital human reasoning model to simulate the operation of the initial digital human reasoning model; during the simulation operation, tracking and recording the computing operations involved in each network layer in the initial digital human reasoning model and the dependencies between the computing operations; and constructing a static computational graph based on the computing operations involved in each network layer and the dependencies between the computing operations.

[0059] In an embodiment of the present application, a control flow analysis is performed on the initial digital human reasoning model to obtain a static calculation graph of the initial digital human reasoning model, including: compiling the forward propagation function in the model file of the initial digital human reasoning model, converting the control flow statements and calculation operations in the forward propagation function into the initial calculation graph; analyzing the dependency relationship between the calculation operations, and adding the dependency relationship to the initial calculation graph to obtain a static calculation graph. Among them, the forward propagation function refers to...

[0060] In an embodiment of the present application, the network structure, weight parameter set and static calculation graph of the initial digital human reasoning model are reorganized according to the target model format to obtain the target digital human reasoning model, including: based on the operator conversion table, the operators in each network layer in the network structure are converted into the target operators in the target model format, and the operators are used to implement the calculation operations involved in the network layer; the operator conversion table stores the correspondence between the original operators in the original model format and the target operators in the target model format; based on the weight parameter conversion relationship, the original weight parameters of each network layer in the weight parameter set are converted into the target weight parameters in the target model format; the weight The heavy parameter conversion relationship stores the correspondence between the original weight parameters in the original model format and the target weight parameters in the target model format; according to the static calculation graph, the network layers in the network structure, the data flow dependencies between the network layers and / or the target operators in the network layers are optimized to obtain the optimized network layers, the data flow dependencies between the network layers and / or the target operations in the network layers as the optimization objects; according to the optimization objects, the target weight parameters and the unoptimized objects, the target digital human reasoning model is generated, and the unoptimized objects refer to the network layers in the network structure, the data flow dependencies between the network layers and the unoptimized objects in the target operators in the network layers.

[0061] In an embodiment of the present application, based on the operator conversion table, the operators in each network layer in the network structure are converted into the target operators under the target model format, including: traversing the original operators in each network layer in the network structure; for the original operator currently traversed, querying the operator conversion table; if the original operator currently traversed is queried in the operator conversion table, replacing the original operator currently traversed with the target operator corresponding to it; if the original operator currently traversed is not queried in the operator conversion table, replacing the original operator currently traversed with a custom operator, and the custom operator supports the target model format.

[0062] In an embodiment of the present application, the initial digital human reasoning model is a neural network model for generating multiple facial area images with continuously changing mouth shapes based on audio feature information; the method also includes: obtaining sample audio data, multiple labeled facial area images with continuously changing mouth shapes corresponding to the sample audio data, and a sample frame rate; converting the sample audio data into sample audio feature information required by the basic digital human reasoning model, and segmenting the sample audio feature information according to the duration and sample frame rate of the sample audio data to obtain multiple sample audio feature fragments; inputting the multiple sample audio feature fragments into the basic digital human reasoning model, mapping the multiple sample audio feature fragments into multiple sample mouth deformation features, and generating multiple sample facial area images with continuously changing mouth shapes based on the multiple sample mouth deformation features; calculating the loss function between the multiple sample facial area images and the multiple labeled facial area images, and if the loss function does not reach the set model training termination condition, adjusting the model parameters of the basic digital human reasoning model until the loss function reaches the set model training termination condition to obtain the initial digital human reasoning model.

[0063] To facilitate understanding, the conversion process of the initial digital human reasoning model is described in detail below with reference to specific examples.

[0064] The conversion process of the digital human reasoning model mainly includes the following steps: model input and analysis: extract the reasoning logic by loading weights and defining the model structure; calculation graph extraction: convert the dynamic graph of PyTorch into a static calculation graph, which is completed through tracing or scripting; operator mapping and weight migration: map the operators of PyTorch to the target framework and convert the weights; calculation graph optimization: optimize the calculation graph to improve the reasoning efficiency; generate the target format model: generate the final model file in the target format such as ONNX, TensorFlow; verification and adjustment: verify whether the converted model is consistent with the original model, and make necessary adjustments. The more specific process is as follows:

[0065] 1. Model input and parsing: When the inference model (such as PyTorch's .pth file) is input into the conversion tool, the first step is to parse and read the model's network structure and weight parameters. In PyTorch, this means loading the model and loading the saved weights using a predefined architecture. Specifically: Use torch.load() to load the .pth file, which will load the model's weight parameters. Next, the computational graph of the model is constructed through a predefined network architecture (usually implemented through the subclass torch.nn.Module). At this stage, the model's reasoning logic (that is, the model's forward propagation forward() function) is extracted, recording the operations of each layer of the model and its data flow.

[0066] 2. Extraction of model computation graph (Tracing or Scripting): In order to convert the model to the target format, the model's computation graph must be converted from a dynamic graph framework (such as PyTorch) to a static graph. This can be done in two ways:

[0067] A. Tracing mode

[0068] In Tracing mode, the conversion tool will track the sequence of operations of the model during forward propagation by feeding the model a set of example inputs. This method can capture the computational graph of the model, but if the model has branching logic or dynamic operations, tracing may not be able to fully capture these dynamic behaviors. For example, control flow operations (such as if-else statements) may be ignored. Specifically, the conversion tool will pass example inputs to the model, trace all tensor operations involved; record the computational operations of each layer (such as convolution, full connection, pooling, etc.) and the corresponding weights; and establish a complete computational graph that describes how to transform from input data through each layer to output data.

[0069] B. Scripting mode

[0070] In Scripting mode, the Python code of the model is converted into a static graph, which can handle more complex dynamic control flows. This method is more suitable for models with complex dynamic logic. The conversion tool analyzes each step of the forward() function and generates a static graph. Specifically, the conversion tool compiles the forward() function of the model, converts the control flow (such as if, for statements) and tensor operations into a static computational graph, analyzes the dependencies of each operation, and generates a complete computational graph for the model.

[0071] 3. Conversion between operators and weights

[0072] The generated computational graph contains the hierarchical structure of the model, the type of operation at each layer (such as convolution, pooling, activation function, etc.), and the data flow dependencies between them. Next, the tool needs to convert these operators and weights into the target format.

[0073] A. Operator Mapping

[0074] Different deep learning frameworks (such as PyTorch, TensorFlow, ONNX) have their own sets of operators. During the conversion process, the tool maps PyTorch's operators to the operators of the target framework according to the requirements of the target format. For example: PyTorch's torch.nn.Conv2d will be mapped to the Conv operator in ONNX; PyTorch's ReLU activation function will be mapped to the Relu operator in ONNX. This mapping is achieved through an operator conversion table. The tool checks the operation type of each layer and looks for the corresponding operator in the target format. If some operations are not supported in the target format, manual adjustments or custom operators may be required.

[0075] B. Weight Migration

[0076] While converting operators, the tool also needs to extract the weight parameters of each layer (such as convolution kernels, biases, etc.) from .pth and convert them to the target format. For example, weights in PyTorch are usually torch.Tensor objects, while in ONNX, weights will be converted to standard multidimensional array format; these weights need to be saved according to the format and requirements of the target framework; the weight migration process ensures that the model can use the same trained parameters for inference in the target framework.

[0077] 4. Optimization of computational graph

[0078] After generating the computational graph in the target format, the conversion tool usually performs a series of optimizations to improve the efficiency of model reasoning. Common optimizations include: Layer merging: For example, convolutional layers and batch normalization (BatchNorm) layers are merged into one operation to reduce the amount of calculation; Constant folding: For some static operations, such as addition and multiplication, the tool may calculate the results in advance during the compilation phase to reduce the computational burden during reasoning; Deleting redundant nodes: If the model contains unused layers or operations, the tool will delete them to optimize the size and speed of the model.

[0079] 5. Generate target format model

[0080] After completing the operator mapping and weight migration, the tool will generate the final model file according to the target format standard. For example, when converting to ONNX, the tool will generate a .onnx file containing the network structure and weights, which can be run in an ONNX-compatible reasoning engine (such as ONNX Runtime); when converting to TensorFlow, it may generate a .pb file or save it in the TensorFlow SavedModel format.

[0081] 6. Model verification and adjustment

[0082] After the conversion is completed, the last step is to verify the consistency of the model. The tool usually verifies whether the model has been successfully converted in the following ways: Run inference verification: Use the same input data to perform inference in the original model and the converted model to check whether the output is consistent; Adjust incompatible layers: If some layers are not fully supported in the target format, the tool may generate warnings and allow developers to make manual adjustments. For example, some special custom layers may need to be rewritten or provide alternative implementations.

[0083] For example, when converting a .pth model to .onnx format, the format conversion of the model's network structure and weights is mainly involved, but the core working principle and structure of the model remain largely unchanged. The following is a detailed description:

[0084] In PyTorch, a .pth file usually contains two parts: model weight parameters: these are all the parameters of the trained model, such as the weights and biases of each layer of the network; network structure: this is the code logic that defines how the model receives input, calculates through different layers, and produces output. When converted to .onnx, the network structure and weights of the model will be converted into the .onnx format. This process preserves the network's computational graph and trained weights. For the conversion of weight parameters, the model's weight parameters are extracted from .pth and saved in .onnx. These parameters will be stored in a format compatible with ONNX. For the conversion of network structure: the structure of the model will also be converted from PyTorch's computational graph representation to the ONNX format. This includes the types of layers (such as convolutional layers, fully connected layers, activation functions, etc.) and how they are connected.

[0085] It should be noted that during the conversion process, the network structure usually does not change, but there are some situations that require attention:

[0086] 1. Layers that are not fully compatible between PyTorch and ONNX: Some special layers or operations implemented in PyTorch may not be directly converted to ONNX format. These layers may be replaced with equivalent layers supported by ONNX. For example, some specific custom layers or operations, such as some uncommon dynamic operations, may require manual adjustments or custom implementations during conversion.

[0087] 2. The difference between static graphs and dynamic graphs: PyTorch is a deep learning framework based on dynamic graphs, which means that the network structure is dynamically constructed at runtime; while ONNX is a static graph, which means that the network structure needs to be fixed during conversion, and all dynamic operations need to be converted into static computational graphs. This conversion may affect some dynamic structures, resulting in the need to rewrite the network structure into a compatible static version.

[0088] 3. Changes in working principles: Usually the basic working principle of the model will not change, but if you encounter incompatible network layers during conversion (dynamic operations as described above), you need to make some adjustments to the model to ensure that it can be correctly inferred in the ONNX runtime (such as ONNX Runtime or other frameworks that support ONNX). 4. The main network layers of the converted model: Convolutional layers: Generally, they will not change and are still used for feature extraction. Fully connected layers: Consistent with PyTorch, they are still used for classification or regression. Activation functions: Most common activation functions (such as ReLU, Sigmoid) will be retained intact. If there is a network involving a dynamic structure, it may be implemented through some static alternatives of ONNX after conversion, and the working principle will be slightly adjusted.

[0089] 4. Notes on conversion: Model input and output: PyTorch may allow multiple forms of input, while ONNX requires explicit input and output tensor shapes. This may require you to fix the input size during the conversion process, or explicitly specify dynamic dimensions. Operations during export: PyTorch torch.onnx.export() usually needs to be called

[0090] method to convert, and when exporting, make sure to provide the correct example input so that the model's calculation graph can be exported correctly.

[0091] 5. It should also be noted that when converting the Wav2Lip .pth model to the .onnx format, the network structure and weight parameters are mainly converted, and the core network structure will not change. If the model contains layers or operations that are incompatible with PyTorch and ONNX, some static substitution adjustments may be made to these layers. In more special cases, manual adjustments may be required based on the specific model and operation to ensure that the network structure in ONNX is consistent with that in PyTorch.

[0092] The embodiment of the present application provides a model conversion process, in which an initial digital human reasoning model adapted to the system type and required model functions of a mobile terminal is determined in a digital human reasoning model developed based on a target model development framework, wherein the target model development framework refers to a model development framework applicable to a server; when the available resource information of the mobile terminal meets the resource loading conditions required by the data packet, a target format conversion tool is determined according to the original model format supported by the initial digital human reasoning model and a model-format conversion tool mapping table, wherein the model-format conversion tool mapping table stores format conversion tools corresponding to digital human reasoning models supporting different model formats; the target format conversion tool is called to convert the model format of the initial digital human reasoning model to obtain a target digital human reasoning model supporting the target model format and suitable for running on a mobile terminal, thereby improving the reasoning speed of the digital human model, and also reducing the delay and freeze problems of the subsequent digital human video, thereby improving the playback quality of the target digital human video.

[0093] Figure 3 The following is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. Figure 3 As shown, it includes: a memory 30a and a processor 30b; the memory 30a is used to store a computer program; the processor 30b is coupled to the memory 30a and is used to execute the computer program to implement the following steps:

[0094] The method comprises: obtaining audio data for generating the voice of a digital human, an image resource package for generating the image of a digital human, and a frame rate of a digital human video to be generated, wherein the image resource package includes multiple original digital human base maps, original facial region images in each digital human base map, and position information of each original facial region image in the corresponding original digital human base map; converting the audio data into target audio feature information required by a target digital human inference model, and performing segmentation operation on the target audio feature information according to the duration and frame rate of the audio data to obtain multiple audio feature segments; inputting the multiple audio feature segments into the target digital human inference model, Mapping multiple audio feature segments into multiple mouth deformation features, and generating multiple target facial region images with continuously changing mouth shapes based on the multiple mouth deformation features; replacing the original facial region images in the multiple original digital human base images with the multiple target facial region images according to the position information of each original facial region image in the corresponding original digital human base image, so as to obtain multiple target digital human images with continuously changing mouth shapes; generating a digital human action picture according to the multiple target digital human images with continuously changing mouth shapes, and driving the digital human mouth in the digital human action picture to broadcast audio data, so as to obtain a target digital human video.

[0095] In an embodiment of the present application, when the processor 30b converts the audio data into the audio feature information required by the target digital human reasoning model, it is specifically used to: extract Fourier transform data from the audio data; generate Mel spectrum data based on the Fourier transform data; and crop, fill or resize the Mel spectrum data according to the requirements of the target digital human reasoning model for the input data to obtain the target audio feature information.

[0096] In an embodiment of the present application, multiple audio feature fragments are input into a target digital human inference model, and the multiple audio feature fragments are mapped into multiple mouth deformation features. When the processor 30b generates multiple target facial area images with continuously changing mouth shapes based on the multiple mouth deformation features, it is specifically used to: input multiple audio feature fragments into the target digital human inference model, and based on the multiple audio feature fragments, infer the changes in the digital human's mouth shape to obtain multiple mouth deformation features; convert the multiple mouth deformation features into multiple pixel point data in the facial area and position distribution information of each pixel point; generate multiple target facial area images according to the multiple pixel point data in the facial area corresponding to the multiple mouth deformation features and the position distribution information of each pixel point.

[0097] In an embodiment of the present application, when the processor 30b infers the changes in the mouth shape of the digital human based on multiple audio feature segments and obtains multiple mouth deformation features, it is specifically used to: generate mouth deformation features corresponding to each audio feature segment, the mouth deformation features are multidimensional feature vectors, and the multidimensional feature vectors at least include mouth shape parameters, opening and closing parameters and lip position parameters; convert the multiple mouth deformation features into multiple pixel point data in the facial area and position distribution information of each pixel point, including: based on the mouth shape parameters, opening and closing parameters and lip position parameters corresponding to each mouth deformation feature, convert each mouth deformation feature into multiple pixel point data in the facial area and position distribution information of each pixel point.

[0098] In an embodiment of the present application, when the processor 30b performs a segmentation operation on the target audio feature information according to the duration and frame rate of the audio data to obtain multiple audio feature segments, it is specifically used to: use the duration of the audio data as the dividend and the frame rate as the divisor to perform a quotient operation to obtain the video frame duration; use the video frame duration as the segmentation unit to segment the target audio feature information to obtain multiple audio feature segments.

[0099] In an embodiment of the present application, the target digital human reasoning model is obtained by converting the model format of the initial digital human reasoning model, and the initial digital human reasoning model is developed on the server side through a target model development framework suitable for the server side.

[0100] Further optionally, the processor 30b is also used to send a model conversion request message to the server, the model conversion request message includes the system type of the mobile terminal, available resource information and model functions required by the mobile terminal, so as to request the server to convert the initial digital human reasoning model into the target digital human reasoning model based on the system type, available resource information and model functions; receive the model conversion response message returned by the server, the model conversion response message is used to notify the mobile terminal to obtain the model file of the target digital human reasoning model from the remote persistent storage space; obtain the model file of the target digital human reasoning model from the remote persistent storage space and store it in the local persistent storage space, and when using the target digital human reasoning model, load the model file from the local persistent storage space to run the target digital human reasoning model.

[0101] Furthermore, if Figure 3 As shown, the server also includes: a communication component 30c, a display 30d, a power component 30e, an audio component 30f and other components. Figure 3 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 3 Components shown.

[0102] The detailed implementation and beneficial effects of the electronic device provided in the embodiments of the present application have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0103] The exemplary embodiments of the present application further provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is caused to implement the steps in the above-mentioned method embodiments.

[0104] An exemplary embodiment of the present application further provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the processor is enabled to implement the steps in the above-mentioned method embodiments.

[0105] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0106] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0107] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0108] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0109] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0110] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage, etc.) containing computer-usable program code.

[0111] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0114] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0115] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0116] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0117] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0118] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for generating digital human video, It is characterized in that Applied to a mobile terminal, the mobile terminal runs a target digital human reasoning model, and the method includes: Acquire audio data for generating digital human voice, an image resource package for generating digital human image, and a frame rate of a digital human video to be generated, wherein the image resource package includes a plurality of original digital human base maps, an original face region image in each digital human base map, and position information of each original face region image in the corresponding original digital human base map; Converting the audio data into target audio feature information required by the target digital human reasoning model, and segmenting the target audio feature information according to the duration of the audio data and the frame rate to obtain multiple audio feature segments; Inputting the multiple audio feature segments into the target digital human inference model, mapping the multiple audio feature segments into multiple mouth deformation features, and generating multiple target facial area images with continuously changing mouth shapes based on the multiple mouth deformation features; According to the position information of each original face region image in the corresponding original digital human base image, the original face region images in the multiple original digital human base images are replaced with the multiple target face region images to obtain multiple target digital human images with continuously changing mouth shapes; A digital human action picture is generated according to the plurality of target digital human images with continuously changing mouth shapes, and the digital human mouth in the digital human action picture is driven to broadcast the audio data to obtain a target digital human video.

2. The method according to claim 1, It is characterized in that Converting the audio data into audio feature information required by the target digital human reasoning model includes: Extracting Fourier transform data from the audio data; generating Mel spectrum data based on the Fourier transform data; and cropping, padding or resizing the Mel spectrum data according to the requirements of the target digital human inference model for input data to obtain the target audio feature information.

3. The method according to claim 1, It is characterized in that Inputting the multiple audio feature segments into the target digital human inference model, mapping the multiple audio feature segments into multiple mouth deformation features, and generating multiple target facial region images with continuously changing mouth shapes based on the multiple mouth deformation features, including: inputting the multiple audio feature segments into the target digital human inference model, and inferring the changes in the digital human's mouth shape based on the multiple audio feature segments to obtain multiple mouth deformation features; Converting the multiple mouth deformation features into multiple pixel point data and position distribution information of each pixel point in the facial area respectively; The multiple target facial region images are generated according to the multiple pixel point data in the facial region corresponding to the multiple mouth deformation features and the position distribution information of each pixel point.

4. The method according to claim 3, It is characterized in that Based on the multiple audio feature segments, the mouth shape changes of the digital human are inferred to obtain multiple mouth deformation features, including: Generate a mouth deformation feature corresponding to each audio feature segment, wherein the mouth deformation feature is a multidimensional feature vector, and the multidimensional feature vector at least includes a mouth shape parameter, an opening and closing degree parameter, and a lip position parameter; The multiple mouth deformation features are converted into multiple pixel point data in the facial area and position distribution information of each pixel point, including: Based on the mouth shape parameters, the opening and closing degree parameters and the lip position parameters corresponding to each mouth deformation feature, each mouth deformation feature is converted into a plurality of pixel point data in a facial area and position distribution information of each pixel point.

5. The method according to claim 1, It is characterized in that According to the duration of the audio data and the frame rate, the target audio feature information is segmented to obtain a plurality of audio feature segments, including: The duration of the audio data is used as the dividend and the frame rate is used as the divisor to perform a quotient operation to obtain the duration of the video frame; The target audio feature information is segmented using the video frame duration as a segmentation unit to obtain a plurality of audio feature segments.

6. The method according to any one of claims 1 to 5, It is characterized in that The target digital human reasoning model is obtained by converting the model format of the initial digital human reasoning model, and the initial digital human reasoning model is developed on the server side through a target model development framework suitable for the server side.

7. The method according to claim 6, It is characterized in that Also includes: Sending a model conversion request message to the server, wherein the model conversion request message includes the system type of the mobile terminal, available resource information, and model functions required by the mobile terminal, so as to request the server to convert the initial digital human reasoning model into the target digital human reasoning model based on the system type, available resource information, and model functions; Receiving a model conversion response message returned by the server, wherein the model conversion response message is used to notify the mobile terminal to obtain a model file of the target digital human reasoning model from a remote persistent storage space; The model file of the target digital human reasoning model is obtained from the remote persistent storage space and stored in the local persistent storage space, and when the target digital human reasoning model is used, the model file is loaded from the local persistent storage space to run the target digital human reasoning model.

8. An electronic device, It is characterized in that include: Memory and processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the processor is caused to implement the steps in the method according to any one of claims 1 to 7.

10. A computer program product, It is characterized in that The computer program product comprises a computer program / instruction, which, when executed by a processor, enables the processor to implement the steps of any one of the methods of claims 1 to 7.

Images