Model conversion methods, devices, storage media, and software products

Abstract

This application provides a model conversion method, device, storage medium, and program product. In this application embodiment, when the available resource information of the mobile terminal meets the resource loading conditions required for the data packet, a target format conversion tool is determined based on the original model format supported by the initial digital human inference model and a model format conversion tool mapping table. The model format conversion tool mapping table stores format conversion tools corresponding to digital human inference models that support different model formats. The target format conversion tool is invoked to perform model format conversion processing on the initial digital human inference model to obtain a target digital human inference model that supports the target model format and is suitable for running on the mobile terminal. This can improve the inference speed of the digital human model and reduce subsequent latency and stuttering issues in the digital human video, thereby improving the playback quality of the target digital human video.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a model conversion method, device, storage medium and program product. Background Technology

[0002] With the continuous development of artificial intelligence, AI digital human technology has been widely applied. Currently, most solutions for generating digital humans rely on the server to complete the inference task, and then transmit the generated AI digital human video to the mobile device.

[0003] However, this approach has the following problems: When the server needs to execute multiple inference tasks simultaneously, it may experience slow response times due to excessive load, resulting in slow inference speed. Additionally, after the server generates the digital human video, it needs to transmit it to the mobile device over the network. In poor network conditions, this can cause delays and stuttering when playing the digital human video on the mobile device. Summary of the Invention

[0004] This application provides a model conversion method, apparatus, storage medium, and program product to improve model inference speed and playback quality of digital human videos.

[0005] This application provides a model conversion method applied to a service terminal. The method includes: receiving a model conversion request message sent by a mobile terminal, the model conversion request message including at least the system type of the mobile terminal, available resource information, and the model functions required by the mobile terminal; determining an initial digital human inference model adapted to the system type and required model functions of the mobile terminal from digital human inference models developed based on a target model development framework, where the target model development framework refers to a model development framework applicable to the server; determining a target format conversion tool based on the original model format supported by the initial digital human inference model and a model format conversion tool mapping table, provided that the available resource information of the mobile terminal meets the resource loading conditions required for the data packet; and calling the target format conversion tool to perform model format conversion processing on the initial digital human inference model to obtain a target digital human inference model that supports the target model format, the target digital human inference model being suitable for running on the mobile terminal.

[0006] The application also provides an electronic device, including: a memory and a processor; the memory for storing a computer program; and the processor, coupled to the memory, for executing the computer program to implement the steps in the above-described digital human-based live streaming method.

[0007] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the steps in the live streaming method based on digital humans.

[0008] This application also provides a computer program product, which includes a computer program / instructions that, when executed by a processor, enable the processor to implement the steps in the above-described live streaming method for digital humans.

[0009] In this embodiment, the digital human inference model developed based on the target model development framework determines an initial digital human inference model that is compatible with the system type and required model functions of the mobile terminal. The target model development framework refers to a model development framework applicable to the server. When the available resource information of the mobile terminal meets the resource loading conditions required for the data packet, a target format conversion tool is determined based on the original model format supported by the initial digital human inference model and a model format conversion tool mapping table. The model format conversion tool mapping table stores format conversion tools corresponding to digital human inference models that support different model formats. The target format conversion tool is called to perform model format conversion processing on the initial digital human inference model to obtain a target digital human inference model that supports the target model format and is suitable for running on the mobile terminal. This improves the inference speed of the digital human model and reduces subsequent latency and stuttering issues in the digital human video, thereby improving the playback quality of the target digital human video. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0011] Figure 1 A schematic flowchart illustrating a model conversion method provided for an exemplary embodiment of this application;

[0012] Figure 2 A flowchart illustrating a target digital human video generation process provided for an exemplary embodiment of this application;

[0013] Figure 3 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0016] To address the existing technical problems, in this embodiment of the application, in the digital human inference model developed based on the target model development framework, an initial digital human inference model adapted to the system type and required model functions of the mobile terminal is determined. The target model development framework refers to a model development framework applicable to the server side. When the available resource information of the mobile terminal meets the resource loading conditions required for the data packet, a target format conversion tool is determined according to the original model format supported by the initial digital human inference model and the model format conversion tool mapping table. The model format conversion tool mapping table stores the format conversion tools corresponding to digital human inference models that support different model formats. The target format conversion tool is called to perform model format conversion processing on the initial digital human inference model to obtain a target digital human inference model suitable for running on the mobile terminal, which supports the target model format. This can improve the inference speed of the digital human model and reduce the latency and stuttering problems of the subsequent digital human video, thereby improving the playback quality of the target digital human video.

[0017] The following describes in detail a solution provided by an embodiment of this application, with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart illustrating a model conversion method provided for an exemplary embodiment of this application. Figure 1 As shown, the method is used on the server side, and the method includes:

[0019] 101. Receive a model conversion request message sent by a mobile terminal. The model conversion request message shall include at least the system type of the mobile terminal, available resource information, and the model functions required by the mobile terminal.

[0020] 102. From the digital human reasoning model developed based on the target model development framework, determine the initial digital human reasoning model that is compatible with the system type and required model functions of the mobile terminal. The target model development framework refers to the model development framework applicable to the server.

[0021] 103. When the available resource information of the mobile terminal meets the resource loading conditions required by the data packet, the target format conversion tool is determined according to the original model format supported by the initial digital human inference model and the model format conversion tool mapping table. The model format conversion tool mapping table stores the format conversion tools corresponding to digital human inference models that support different model formats.

[0022] 104. Call the target format conversion tool to convert the initial digital human inference model to obtain a target digital human inference model that supports the target model format. The target digital human inference model is suitable for running on mobile terminals.

[0023] In this application embodiment, the system type includes, but is not limited to, Android, iOS, or HarmonyOS. Available resource information includes, but is not limited to,: CPU information, such as model and speed, number of cores and threads, and cache size; memory (RAM) information, such as total memory capacity, used and available memory, memory speed, and memory timings; storage information, such as hard disk or solid-state drive capacity, available space, read / write speed, and interface type (e.g., SATA, NVMe); graphics processing unit (GPU) information, such as model, video memory size, and GPU utilization; network information, such as network adapter type, IP address, network bandwidth, and network connection status; and hardware status, such as device health status and fault information.

[0024] In this embodiment of the application, determining an initial digital human reasoning model adapted to the system type and required model functions of a mobile terminal from digital human reasoning models developed based on the target model development framework includes: selecting a first candidate digital human reasoning model adapted to the system type of the mobile terminal from the digital human reasoning models developed based on the target model development framework, wherein different digital human reasoning models are configured with their adapted operating system lists; determining a second candidate digital human reasoning model adapted to the required model functions of the mobile terminal from the functional description information of the first candidate digital human reasoning model; if there is only one second candidate digital human reasoning model, using the second candidate digital human reasoning model as the initial digital human reasoning model; if there are multiple second candidate digital human reasoning models, selecting the one with the smallest model file as the initial digital human reasoning model.

[0025] Optionally, the model file of the initial digital human inference model is obtained; based on the size of the model file, it is determined whether the bandwidth information and / or memory information in the available resource information of the mobile terminal meet the minimum bandwidth value and / or minimum memory amount required in the resource loading conditions; if so, it is determined that the available resource information of the mobile terminal meets the resource loading conditions.

[0026] In this embodiment, a target format conversion tool is invoked to convert the initial digital human inference model to a target digital human inference model that supports the target model format. This includes: invoking the target format conversion tool to parse the model file of the initial digital human inference model to obtain the network structure and weight parameter set of the initial digital human inference model. The network structure describes the multiple network layers included in the initial digital human inference model and the connection relationships between the multiple network layers. The weight parameter set includes the original weight parameters of each network layer. The weight parameters represent the degree of influence of the input data on the model output. Specifically, the weights determine the degree to which the input data should be amplified or reduced before being passed to the next layer. Furthermore, by simulating or performing control flow analysis on the initial digital human inference model, a static computation graph of the initial digital human inference model is obtained. The static computation graph includes the processing logic from input data to output data for each network layer and the data flow dependencies between network layers. Based on the target model format, the network structure, weight parameter set, and static computation graph of the initial digital human inference model are reorganized to obtain the target digital human inference model.

[0027] In this embodiment of the application, the model file of the initial digital human inference model is parsed to obtain the network structure and weight parameter set of the initial digital human inference model. This includes: parsing a sub-file with a specific format in the model file to obtain the weight parameter set of the initial digital human inference model; and defining the model structure of the initial digital human inference model based on the forward propagation function in the model file to obtain the network structure of the initial digital human inference model.

[0028] In this embodiment of the application, a static computation graph of the initial digital human inference model is obtained by simulating the initial digital human inference model. This includes: inputting preset exemplary input data into the initial digital human inference model to simulate the initial digital human inference model; during the simulation, tracking and recording the computational operations involved in each network layer of the initial digital human inference model and the dependencies between the computational operations; and constructing a static computation graph based on the computational operations involved in each network layer and the dependencies between the computational operations.

[0029] In this embodiment, a static computation graph of the initial digital human inference model is obtained by performing control flow analysis on the initial digital human inference model. This includes: compiling the forward propagation function in the model file of the initial digital human inference model to transform the control flow statements and computation operations in the forward propagation function into an initial computation graph; analyzing the dependencies between computation operations and adding the dependencies to the initial computation graph to obtain a static computation graph. Here, the forward propagation function refers to...

[0030] In this embodiment, the network structure, weight parameter set, and static computation graph of the initial digital human inference model are reorganized according to the target model format to obtain the target digital human inference model. This includes: converting operators in each network layer of the network structure into target operators in the target model format based on an operator conversion table; the operators are used to implement computational operations involved in the network layers; the operator conversion table stores the correspondence between the original operators in the original model format and the target operators in the target model format; and converting the original weight parameters of each network layer in the weight parameter set into target weight parameters in the target model format based on the weight parameter conversion relationship. The parameter transformation relationship stores the correspondence between the original weight parameters in the original model format and the target weight parameters in the target model format; based on the static computation graph, optimization operations are performed on the network layers, the data flow dependencies between network layers, and / or the target operators in the network layers to obtain the optimized network layers, the data flow dependencies between network layers, and / or the target operators in the network layers, which serve as the optimization objects; based on the optimization objects, the target weight parameters, and the unoptimized objects, the target digital human inference model is generated. The unoptimized objects refer to the network layers, the data flow dependencies between network layers, and the unoptimized objects in the target operators of the network layers.

[0031] In this embodiment, based on the operator conversion table, operators in each network layer of the network structure are converted into target operators in the target model format. This includes: traversing the original operators in each network layer of the network structure; for the currently traversed original operator, querying the operator conversion table; if the currently traversed original operator is found in the operator conversion table, replacing the currently traversed original operator with its corresponding target operator; if the currently traversed original operator is not found in the operator conversion table, replacing the currently traversed original operator with a custom operator, wherein the custom operator supports the target model format.

[0032] In this embodiment, the initial digital human inference model is a neural network model for generating multiple facial region images with continuously changing mouth shapes based on audio feature information. The method further includes: acquiring sample audio data, multiple labeled facial region images with continuously changing mouth shapes corresponding to the sample audio data, and sample frame rate; converting the sample audio data into sample audio feature information required by the basic digital human inference model, and segmenting the sample audio feature information according to the duration and sample frame rate of the sample audio data to obtain multiple sample audio feature segments; inputting the multiple sample audio feature segments into the basic digital human inference model, mapping the multiple sample audio feature segments into multiple sample mouth deformation features, and generating multiple sample facial region images with continuously changing mouth shapes based on the multiple sample mouth deformation features; calculating the loss function between the multiple sample facial region images and the multiple labeled facial region images, and adjusting the model parameters of the basic digital human inference model until the loss function reaches the set model training termination condition if the loss function does not reach the set model training termination condition, so as to obtain the initial digital human inference model.

[0033] To facilitate understanding, the conversion process of the initial digital human reasoning model will be described in detail below with specific examples.

[0034] The conversion process of the digital human inference model mainly includes the following steps: Model input and parsing: Extracting inference logic by loading weights and defining the model structure; Computation graph extraction: Converting the PyTorch dynamic graph into a static computation graph through tracing or scripting; Operator mapping and weight transfer: Mapping PyTorch operators to the target framework and converting weights; Computation graph optimization: Optimizing the computation graph to improve inference efficiency; Generating the target format model: Generating the final model file in target formats such as ONNX and TensorFlow; Verification and adjustment: Verifying whether the converted model is consistent with the original model and making necessary adjustments. A more detailed process is as follows:

[0035] 1. Model Input and Parsing: When an inference model (e.g., a PyTorch .pth file) is input into a conversion tool, the first step is to parse and read the model's network structure and weight parameters. In PyTorch, this means loading the model and using a predefined architecture to load the saved weights. Specifically: using `torch.load()` to load the .pth file loads the model's weight parameters. Next, the model's computation graph is constructed using a predefined network architecture (usually implemented through the subclass `torch.nn.Module`). At this stage, the model's inference logic (i.e., the model's forward propagation function `forward()`) is extracted, recording the operations and data flow of each layer of the model.

[0036] 2. Extraction of the model's computational graph (Tracing or Scripting): To convert the model to the target format, the model's computational graph must be converted from a dynamic graph framework (such as PyTorch) to a static graph. This can be done in two ways:

[0037] A. Tracing mode

[0038] In tracing mode, the transformation tool tracks the sequence of operations performed by the model during forward propagation by feeding it a set of example inputs. This method captures the model's computational graph, but if the model has branching logic or dynamic operations, tracing may not fully capture these dynamic behaviors. For example, control flow operations (such as if-else statements) may be ignored. Specifically, the transformation tool passes example inputs to the model, tracks all tensor operations involved, records the computational operations at each layer (such as convolution, fully connected layers, pooling, etc.) and their corresponding weights, and builds a complete computational graph that describes how the input data is transformed into output data through each layer.

[0039] B. Scripting Mode

[0040] In Scripting mode, the model's Python code is transformed into a static graph, enabling the handling of more complex dynamic control flows. This approach is more suitable for models with complex dynamic logic. The transformation tool analyzes each operation in the `forward()` function and generates a static graph. Specifically, the tool compiles the model's `forward()` function, transforming its control flow (such as `if` and `for` statements) and tensor operations into a static computation graph; it also analyzes the dependencies between operations to generate the model's complete computation graph.

[0041] 3. Conversion between operators and weights

[0042] The generated computation graph contains the model's hierarchical structure, the types of operations at each layer (e.g., convolution, pooling, activation functions), and the data flow dependencies between them. Next, the tool needs to convert these operators and weights into the target format.

[0043] A. Operator Mapping

[0044] Different deep learning frameworks (such as PyTorch, TensorFlow, and ONNX) have their own operator sets. During the conversion process, the tool maps PyTorch operators to operators of the target framework according to the requirements of the target format. For example, PyTorch's `torch.nn.Conv2d` will be mapped to the `Conv` operator in ONNX; PyTorch's ReLU activation function will be mapped to the ReLU operator in ONNX. This mapping is achieved through an operator transformation table. The tool checks the operation type of each layer and looks up the corresponding operator in the target format. If some operations are not supported in the target format, manual adjustments or the use of custom operators may be necessary.

[0045] B. Weight Transfer

[0046] While performing operator transformations, the tool also needs to extract the weight parameters (such as convolution kernels and biases) of each layer from the .pth file and convert them to the target format. For example, weights in PyTorch are usually torch.Tensor objects, while in ONNX, weights are converted into a standard multidimensional array format. These weights need to be saved according to the format and requirements of the target framework. The weight transfer process ensures that the model can use the same trained parameters for inference in the target framework.

[0047] 4. Optimization of the computation graph

[0048] After generating the computation graph in the target format, the conversion tool typically performs a series of optimizations to improve model inference efficiency. Common optimizations include: layer merging: for example, merging convolutional layers and batch normalization (BatchNorm) layers into a single operation to reduce computational cost; constant folding: for some static operations, such as addition and multiplication, the tool may pre-calculate the results during compilation to reduce the computational burden during inference; and redundant node removal: if the model contains unused layers or operations, the tool will remove them to optimize model size and speed.

[0049] 5. Generate the target format model

[0050] After completing operator mapping and weight transfer, the tool generates the final model file according to the target format standard. For example, when converting to ONNX, the tool generates an .onnx file containing the network structure and weights, which can run in ONNX-compatible inference engines (such as ONNX Runtime); when converting to TensorFlow, it may generate a .pb file or save it in TensorFlow SavedModel format.

[0051] 6. Model Validation and Adjustment

[0052] After the conversion is complete, the final step is to verify the consistency of the model. Tools typically verify successful conversion in the following ways: running inference verification: performing inference on both the original and converted models using the same input data to check for consistent output; adjusting incompatible layers: if some layers are not fully supported in the target format, the tool may generate warnings and allow developers to make manual adjustments. For example, certain custom layers may need to be rewritten or alternative implementations provided.

[0053] For example, converting a .pth model to .onnx format mainly involves changing the format of the model's network structure and weights, but the core working principle and structure of the model remain largely unchanged. The following is a detailed explanation:

[0054] In PyTorch, .pth files typically contain two parts: model weight parameters: these are all the parameters of the trained model, such as the weights and biases of each layer; and network structure: this is the code logic that defines how the model receives input, processes it through different layers, and produces output. When converting to .onnx, both the model's network structure and weights are converted to .onnx format. This process preserves the network's computational graph and the trained weights. For the weight parameter conversion, the model's weight parameters are extracted from the .pth file and stored in the .onnx file. These parameters are stored in a format compatible with ONNX. For the network structure conversion, the model's structure is also converted from PyTorch's computational graph representation to ONNX format. This includes the types of each layer (e.g., convolutional layers, fully connected layers, activation functions, etc.) and the connections between them.

[0055] It should be noted that the network structure usually remains unchanged during the conversion process, but there are some situations that require attention:

[0056] 1. Layers not fully compatible with PyTorch and ONNX: Certain special layers or operations implemented in PyTorch may not be directly convertible to ONNX format. These layers may be replaced with equivalent layers supported by ONNX. For example, some specific custom layers or operations, such as certain uncommon dynamic operations, may require manual adjustments or custom implementations during conversion.

[0057] 2. The difference between static and dynamic graphs: PyTorch is a deep learning framework based on dynamic graphs, meaning the network structure is dynamically built at runtime; while ONNX is a static graph, meaning the network structure needs to be fixed during transformation, and all dynamic operations need to be converted into a static computation graph. This transformation may affect some dynamic structures, leading to the need to rewrite the network structure into a compatible static version.

[0058] 3. Changes in Working Principle: The basic working principle of the model usually remains unchanged. However, if incompatible network layers are encountered during conversion (as mentioned above regarding dynamic operations), some adjustments need to be made to the model to ensure correct inference during ONNX runtime (such as ONNX Runtime or other frameworks supporting ONNX). 4. Main Network Layers of the Converted Model: Convolutional Layers (Conv layers): Generally unchanged, still used for feature extraction. Fully Connected Layers: Consistent with PyTorch, still used for classification or regression. Activation Functions: Most common activation functions (such as ReLU, Sigmoid) are preserved unchanged. If the network involves dynamic structures, it may be implemented using some static alternatives in ONNX after conversion, with slight adjustments to the working principle.

[0059] 4. Conversion Considerations: Model Input / Output: PyTorch may allow multiple input formats, while ONNX requires explicit input / output tensor shapes. This may require fixing the input size during conversion or explicitly specifying dynamic dimensions. Export Operations: Typically, this involves calling PyTorch torch.onnx.export().

[0060] The method is used for transformation. When exporting, you must ensure that the correct example input is provided so that the computational graph of the model can be exported correctly.

[0061] 5. It's also important to note that when converting a Wav2Lip .pth model to .onnx format, the main changes are to the network structure and weight parameters; the core network structure remains largely unchanged. If the model contains layers or operations incompatible with PyTorch and ONNX, some static substitutions may be necessary. In more special cases, manual adjustments may be required based on the specific model and operations to ensure the network structure behaves consistently in ONNX compared to PyTorch.

[0062] This application provides a model conversion process. Based on a target model development framework, an initial digital human inference model is determined that is compatible with the system type and required model functions of the mobile terminal. The target model development framework refers to a model development framework suitable for the server. When the available resource information of the mobile terminal meets the resource loading conditions required for the data packet, a target format conversion tool is determined according to the original model format supported by the initial digital human inference model and a model format conversion tool mapping table. The model format conversion tool mapping table stores format conversion tools corresponding to digital human inference models that support different model formats. The target format conversion tool is called to perform model format conversion processing on the initial digital human inference model to obtain a target digital human inference model that supports the target model format and is suitable for running on the mobile terminal. This can improve the inference speed of the digital human model and reduce subsequent latency and stuttering issues in the digital human video, thereby improving the playback quality of the target digital human video.

[0063] The following is combined Figure 2 The generation process of the target digital human video is described in detail.

[0064] S1. Obtain audio data for generating digital human voice, image resource package for generating digital human image, and frame rate of digital human video to be generated. The image resource package contains multiple original digital human base maps, original facial region images in each digital human base map, and position information of each original facial region image in the corresponding original digital human base map.

[0065] S2. Convert the audio data into target audio feature information required by the target digital human inference model, and perform segmentation operation on the target audio feature information according to the duration and frame rate of the audio data to obtain multiple audio feature segments.

[0066] S3. Input multiple audio feature segments into the target digital human inference model, map the multiple audio feature segments into multiple mouth deformation features, and generate multiple target facial region images with continuously changing mouth shapes based on the multiple mouth deformation features.

[0067] S4. Based on the position information of each original facial region image in the corresponding original digital human base image, replace the original facial region images in multiple original digital human base images with multiple target facial region images to obtain multiple target digital human images with continuously changing mouth shapes.

[0068] S5. Based on multiple images of the target digital human whose mouth shape changes continuously, generate a digital human motion picture, and drive the digital human's voice audio data in the digital human motion picture to obtain the target digital human video.

[0069] In this application embodiment, the implementation form of the mobile terminal is not limited. The mobile terminal can be a smart handheld device, such as a smartphone, tablet computer, laptop computer, or desktop computer, etc.; or, the mobile terminal can also be a smart wearable device, such as a smartwatch, smart bracelet, etc.; or, the mobile terminal can also be various smart home appliances with display screens, such as smart TVs, smart large screens, or smart robots, etc.

[0070] In this embodiment, audio data refers to the audio data containing speech content. This audio data can be pre-recorded or generated in advance or in real-time by the server or mobile terminal based on text information. The timing of audio data generation depends on the specific application scenario, and this embodiment does not impose specific limitations on this. For example, the real-time flexible dialogue audio data in an AI interview room can be audio data used for responses generated in real-time based on the text information corresponding to the input content of the interlocutors, while the audio data corresponding to fixed phrases in some dialogue scenarios can be pre-recorded or generated in advance based on text information.

[0071] To determine the deformation of the mouth in response to audio data, this embodiment introduces a digital human inference model. This model is at least an AI model used to infer mouth shapes based on audio data. Specifically, it can predict and generate lip-shape animations synchronized with the input audio data. For example, it can be used to generate realistic animations of digital humans or virtual characters, allowing the characters to move their mouths naturally according to the audio data, thus improving the accuracy of mouth deformation recognition. This embodiment does not limit the type of digital human inference model. For example, the digital human inference model can be Wav2Lip, ER-NeRF, VideoRetalking, MuseTalk, EchoMimic, etc. Wav2Lip is an open-source lip-sync tool that can generate matching lip-shape animations based on input speech. It is applicable to any face and any language and can generate realistic lip movements. ER-NeRF is a real-time Talking Portrait Synthesis model that can synchronize given audio with the mouth shapes of people in a video. It should be noted that specific videos need to be trained to generate the corresponding digital human inference model.

[0072] VideoRetalking enables synchronization of lip movements in video characters with the input audio. This model includes face video generation, audio-driven lip-sync, and facial enhancement. MuseTalk is a real-time audio-driven lip-sync model that supports multiple languages ​​and is particularly suitable for high-definition facial areas. EchoMimic imbues static images with vivid speech and expressions, creating highly realistic dynamic portrait videos by combining audio and facial landmarks through a deep learning model. These models analyze the input speech signal to generate corresponding lip-sync animations, ensuring that the lip movements of the digital human character match the actual speech content. This improves the accuracy of lip deformation recognition while enhancing the realism of the character and the viewer's immersion. For ease of distinction and description, the actual applied digital human reasoning model is referred to as the target digital human reasoning model. This embodiment focuses on the Wav2Lip model.

[0073] It should be noted that the target digital human inference model is obtained by converting the initial digital human inference model into a different format. The initial digital human inference model is developed on the server using a target model development framework suitable for the server. Specifically, when the mobile terminal requires the target digital human model, it sends a model conversion request message to the server. This message includes the mobile terminal's system type, available resource information, and the model functions required by the mobile terminal, requesting the server to convert the initial digital human inference model into the target digital human inference model based on the system type, available resource information, and model functions. Further, the mobile terminal receives a model conversion response message from the server, which instructs it to retrieve the model file of the target digital human inference model from a remote persistent storage space. The model file is then retrieved from the remote persistent storage space and stored in the local persistent storage space. When using the target digital human inference model, the model file is loaded from the local persistent storage space to run the target digital human inference model.

[0074] It should also be noted that the development process of the initial digital human model may or may not include a training process. If the development process does not include a training process, the training process can be completed on the server before model format conversion, on the mobile terminal before model format conversion, or on the mobile terminal after model format conversion. The development, training, and model format conversion processes of the initial digital human inference model can be found in the relevant descriptions of the following embodiments, and will not be repeated here.

[0075] In this embodiment, the specific implementation of the target digital human inference model running on the mobile terminal is not limited. For example, the mobile terminal can directly run the target digital human inference model; or, the target digital human inference model can run on an application on the mobile terminal. For ease of description and distinction, this application is referred to as the target application. Furthermore, this embodiment does not limit the implementation form of the target application. The target application can be a standalone app, a webpage, or a mini-program that depends on an app for operation.

[0076] In this embodiment, to obtain a digital human video, audio data for generating the digital human's voice and an image resource package for generating the digital human's image can be acquired first. This allows for inference of lip shape changes corresponding to the digital human image based on the audio data, and further, based on the audio data and lip shape changes, a digital human video can be obtained. The image resource package may contain only multiple original digital human base images, or it may simultaneously contain multiple original digital human base images, original facial region images from each base image, and position information of each original facial region image within its corresponding base image. This embodiment does not limit this. Multiple original digital human base images may contain multiple sets of original digital human base images. Each set of original digital human base images can be multiple consecutive digital human images extracted from any digital human video, and the lip shape in these consecutive digital human images changes continuously.

[0077] The original audio data format may differ from the format required by the target digital human inference model. Therefore, after acquiring the audio data, to facilitate the target digital human inference model's reasoning about lip movements corresponding to the audio data, the original audio data format can be converted to the format required by the target digital human inference model. This involves converting the initial audio feature information of the audio data into the target audio feature information required by the target digital human inference model. It should be noted that different digital human inference models may require the same or different types of audio feature information. Taking the target digital human inference model as a Wav2Lip model as an example, the original audio data format is .wav, while the Wav2Lip model requires Mel-spectral data format. Therefore, the audio data needs to be converted from .wav format to Mel-spectral data format, that is, the initial audio feature information of the audio data is converted into Mel-spectral data required by the Wav2Lip model. Mel-spectral data corresponds to a Mel-spectral graph.

[0078] In this embodiment, converting audio data into audio feature information required by the target digital human inference model includes: extracting Fourier transform data from the audio data; generating Mel spectrum data based on the Fourier transform data; and cropping, padding, or resizing the Mel spectrum data according to the requirements of the target digital human inference model for the input data to obtain the target audio feature information. Cropping refers to removing edge noise from the Mel spectrum data or retaining the required spectral portion. Padding refers to adding zero values ​​or other specific values ​​to the edges of the data; typically, when the size of the Mel spectrum data does not meet the model's input requirements, padding can be used to adjust the data size to meet the requirements. Resizing refers to resampling or interpolation to change the resolution or size of the Mel spectrum data to adapt it to a specific input format. The target audio feature information refers to the Mel spectrum feature information corresponding to the Mel spectrum data processed based on the requirements of the target digital human inference model's input data.

[0079] In one optional embodiment, Fourier transform data is extracted from the audio data, and spectral data is generated based on the Fourier transform data. This includes: first, loading the audio data using an appropriate library, such as using the Librosa library to load .wav format audio data; preprocessing the audio data, including but not limited to normalizing volume and removing noise, to improve the accuracy of subsequent processing; converting the audio data into an audio signal, which corresponds to key feature information; performing a Fast Fourier Transform (FFT) on the audio signal to convert it from the time domain to the frequency domain; and using a Mel filter to convert the audio signal in the frequency domain to the Mel scale, generating a Mel spectrogram, which corresponds to Mel spectral data. Further, a Discrete Cosine Transform (DCT) can be performed on the Mel spectrogram to extract Mel Frequency Cepstral Coefficients (MFCCs). The Mel Frequency Cepstral Coefficients are a characteristic representation of the Mel spectral data, i.e., the Mel spectral coefficients are the Mel spectral feature information corresponding to the Mel spectral data.

[0080] Alternatively, in order to improve the accuracy of the inference of the target digital human inference model, the target audio feature information can be segmented, and the target digital human inference model can be used to infer each segment of audio feature information separately to obtain multiple digital human images with continuously changing mouth shapes.

[0081] In one optional embodiment, the target feature information can be segmented based on the duration of the audio data and the frame rate of the digital human video to be generated. The frame rate refers to the number of static images (frames) that can be displayed per second during video playback. Frame rate is an important indicator of video playback smoothness and is usually measured in "Frames Per Second" (FPS). For example, if the frame rate of the digital human to be generated is 25fps, then 25 images will be played within 1 second. Based on this, the segmentation operation of the target audio feature information includes: obtaining the frame rate of the digital human video to be generated; and segmenting the target feature information according to the duration of the audio data and the frame rate of the digital human video to be generated, resulting in multiple audio feature segments.

[0082] When determining the frame rate for the digital human video to be generated, factors such as the intended use of the video, the viewers' devices, the playback platform, and the desired visual effects need to be considered. For example, if the digital human video is intended for online playback and needs to display smooth facial expressions and synchronized lip movements, a frame rate of 30fps or higher can be selected; if the digital human video is intended for film or art projects, 24fps can be selected to suit the viewing experience. Furthermore, technical limitations and costs must be considered. Higher frame rates typically mean processing more data during editing and rendering, which may increase the demand for computing resources and costs. Simultaneously, the size of high frame rate digital human video files will also increase, potentially affecting storage and transmission efficiency. In practical applications, when used in cloud-based intelligent digital human services, the video frame rate is generally no less than 25fps and no more than 60fps; when used for silicon-based digital human cloning, the video frame rate is generally 30fps or 60fps, thus ensuring smoothness and detail while also considering the technical and cost limitations of practical applications.

[0083] Optionally, based on the duration and frame rate of the audio data, the target audio feature information is segmented to obtain multiple audio feature segments. This includes: performing a quotient operation by using the audio data duration as the divisor and the frame rate as the divisor to obtain the video frame duration, where the video frame duration represents the duration of each video frame; and using the video frame duration as the segmentation unit to segment the target audio feature information to obtain multiple audio feature segments. For example, if the audio data duration is 50 seconds and the frame rate of the digital human video to be generated is 25 fps, then...

[0084] Furthermore, after obtaining multiple audio feature segments, these segments can be input into the target digital human inference model to map them into multiple mouth deformation features. Based on these mouth deformation features, multiple target facial region images with continuously changing mouth shapes can be generated.

[0085] In this embodiment, multiple audio feature segments are input into a target digital human inference model, and the multiple audio feature segments are mapped into multiple mouth deformation features. Based on the multiple mouth deformation features, multiple target facial region images with continuously changing mouth shapes are generated. The process includes: inputting multiple audio feature segments into the target digital human inference model; inferring changes in the mouth shape of the digital human based on the multiple audio feature segments to obtain multiple mouth deformation features; converting the multiple mouth deformation features into multiple pixel data and positional distribution information of each pixel in the facial region; and generating multiple target facial region images based on the multiple pixel data and positional distribution information of each pixel in the facial region corresponding to the multiple mouth deformation features.

[0086] In one optional embodiment, the target digital human inference model includes an audio feature processing network layer, which infers mouth shape changes based on multiple audio feature segments to obtain multiple mouth deformation features. This includes inputting multiple audio feature segments into the audio feature processing network layer to generate mouth deformation features corresponding to each audio feature segment. Each mouth deformation feature is a multi-dimensional feature vector, which includes at least a mouth shape parameter, an opening / closing parameter, and a lip position parameter. The mouth shape parameter represents the current mouth shape, the opening / closing parameter represents the current mouth opening / closing degree, and the lip position parameter represents the current mouth position within the face region.

[0087] Optionally, the target digital human inference model may further include: a visual feature processing network layer, which converts multiple mouth deformation features into multiple pixel data and positional distribution information of each pixel in the face region, including: inputting multiple mouth deformation features into the video feature processing network layer, and converting each mouth deformation feature into multiple pixel data and positional distribution information of each pixel in the face region based on the mouth shape parameters, opening and closing parameters and lip position parameters corresponding to each mouth deformation feature.

[0088] Optionally, the target digital human inference model also includes a generator. After obtaining multiple pixel data and the positional distribution information of each pixel, the multiple pixel data and the positional distribution information of each pixel can be input into the generator. Based on the multiple pixel data and the positional distribution information of each pixel, multiple target facial region images with continuously changing mouth shapes are obtained.

[0089] As can be seen from the above embodiments, the image resource package contains multiple original digital human base images, original facial region images in each digital human base image, and position information of each original facial region image in the corresponding original digital human base image. After obtaining multiple target facial region images with continuously changing mouth shapes, the original facial region images in the multiple original digital human base images can be replaced with the multiple target facial region images according to the position information of each original facial region image in the corresponding original digital human base image, so as to obtain multiple target digital human images with continuously changing mouth shapes.

[0090] Furthermore, after obtaining multiple images of the target digital human with continuously changing mouth shapes, a video of the target digital human can be obtained based on these images. This embodiment does not limit the specific implementation method of obtaining a video of the target digital human based on multiple images of the target digital human with continuously changing mouth shapes; the implementation method may vary depending on the application scenario.

[0091] In one optional embodiment, obtaining a target digital human video based on multiple target digital human images showing continuous changes in mouth shape includes: generating digital human motion frames based on the multiple target digital human images showing continuous changes in mouth shape, and driving a digital human in the digital human motion frames to verbally play the audio data to obtain the target digital human video. In this target digital human video, the digital human motion frames and the verbally played audio are two independent parts, suitable for real-time dialogue scenarios, such as real-time question-and-answer sessions in AI live streaming rooms / interview rooms.

[0092] In another optional embodiment, a target digital human video is obtained based on multiple target digital human images with continuously changing mouth shapes, including: generating the target digital human video based on the multiple target digital human images with continuously changing mouth shapes and audio data used to generate the digital human's voice. In this target digital human video, the digital human's motion visuals and spoken audio are a unified whole, suitable for non-real-time playback scenarios, such as non-real-time live-streamed digital human videos. In this embodiment, correspondingly, the generator in the target digital human inference model can also obtain the target digital human video based on multiple target digital human images with continuously changing mouth shapes. Further, the target digital human inference model can also include a lip-sync discriminator for evaluating the synchronization accuracy between the generated video frames and audio data. If the evaluation result does not meet the synchronization accuracy threshold, the target digital human inference model can be retrained based on the synchronization accuracy standard, and the retrained target digital human inference model can be used to generate the target digital human video based on the multiple target digital human images with continuously changing mouth shapes and audio data used to generate the digital human's voice. Furthermore, the target digital human inference model can also include a visual quality discriminator to evaluate whether the generated video frames look natural and without obvious generation artifacts. Similarly, if the evaluation results do not meet the synchronization accuracy threshold, the target digital human inference model can be retrained based on the synchronization accuracy standard. The retrained target digital human inference model can then be used to generate a target digital human video based on multiple target digital human images with continuously changing mouth shapes and audio data used to generate the digital human's voice.

[0093] In the embodiments described above, the original inference environment of the target digital human inference model is changed from inference execution on the server side to inference on the mobile terminal. This avoids the problem that the server may experience slow inference speed due to excessive load when multiple inference tasks need to be executed simultaneously, thus improving the inference speed of the target digital human model. Furthermore, by using the target digital human inference model to infer multiple target facial region images with continuously changing mouth shapes based on multiple audio feature segments corresponding to audio data, and generating digital human motion frames based on these multiple images of continuously changing mouth shapes, the digital human inference frames drive the digital human to speak audio data, thereby obtaining the target digital human video. This not only improves the adaptation between mouth shape and audio content but also reduces latency and stuttering issues when playing the digital human video, improving the playback quality of the target digital human video.

[0094] In the digital human video generation process provided in the application embodiment, the initial digital human inference model is a neural network model used to generate multiple facial region images with continuously changing mouth shapes based on audio feature information; the method further includes: acquiring sample audio data, multiple labeled facial region images with continuously changing mouth shapes corresponding to the sample audio data, and sample frame rate; converting the sample audio data into sample audio feature information required by the basic digital human inference model, and performing segmentation operations on the sample audio feature information according to the duration and sample frame rate of the sample audio data to obtain multiple sample audio feature segments; inputting the multiple sample audio feature segments into the basic digital human inference model, mapping the multiple sample audio feature segments into multiple sample mouth deformation features, and generating multiple sample facial region images with continuously changing mouth shapes based on the multiple sample mouth deformation features; calculating the loss function between the multiple sample facial region images and the multiple labeled facial region images, and adjusting the model parameters of the basic digital human inference model until the loss function reaches the set model training termination condition if the loss function does not reach the set model training termination condition, so as to obtain the initial digital human inference model.

[0095] Figure 3 This is a schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application. For example... Figure 3 As shown, it includes: a memory 30a and a processor 30b; the memory 30a is used to store computer programs; the processor 30b, coupled to the memory 30a, is used to execute the computer programs to achieve the following steps:

[0096] The system receives a model conversion request message from a mobile terminal. This message includes at least the mobile terminal's system type, available resource information, and the model functions required by the mobile terminal. From the digital human inference models developed based on the target model development framework, an initial digital human inference model adapted to the mobile terminal's system type and required model functions is determined. The target model development framework refers to a model development framework suitable for the server. If the mobile terminal's available resource information meets the resource loading conditions required for the data packet, a target format conversion tool is determined based on the original model format supported by the initial digital human inference model and a model format conversion tool mapping table. The model format conversion tool mapping table stores format conversion tools corresponding to digital human inference models supporting different model formats. The target format conversion tool is then invoked to perform model format conversion processing on the initial digital human inference model to obtain a target digital human inference model that supports the target model format and is suitable for running on the mobile terminal.

[0097] In this embodiment, when processor 30b determines an initial digital human reasoning model adapted to the system type and required model functions of a mobile terminal from digital human reasoning models developed based on the target model development framework, it specifically performs the following steps: selecting a first candidate digital human reasoning model adapted to the system type of the mobile terminal from the digital human reasoning models developed based on the target model development framework, wherein different digital human reasoning models are configured with their own list of adapted operating systems; determining a second candidate digital human reasoning model adapted to the required model functions of the mobile terminal from the functional description information of the first candidate digital human reasoning model; if there is only one second candidate digital human reasoning model, using the second candidate digital human reasoning model as the initial digital human reasoning model; if there are multiple second candidate digital human reasoning models, selecting the one with the smallest model file as the initial digital human reasoning model.

[0098] Optionally, the processor 30b is further configured to acquire the model file of the initial digital human inference model; based on the size of the model file, determine whether the bandwidth information and / or memory information in the available resource information of the mobile terminal meets the minimum bandwidth value and / or minimum memory amount required in the resource loading conditions; if so, determine that the available resource information of the mobile terminal meets the resource loading conditions.

[0099] In this embodiment, when processor 30b calls a target format conversion tool to convert the initial digital human inference model to a target digital human inference model that supports the target model format, it specifically performs the following steps: It calls the target format conversion tool to parse the model file of the initial digital human inference model to obtain the network structure and weight parameter set of the initial digital human inference model. The network structure describes the multiple network layers included in the initial digital human inference model and the connection relationships between the multiple network layers. The weight parameter set includes the original weight parameters of each network layer. It also performs simulation operation or control flow analysis on the initial digital human inference model to obtain the static computation graph of the initial digital human inference model. The static computation graph includes the processing logic from input data to output data for each network layer and the data flow dependencies between network layers. Finally, it reorganizes the network structure, weight parameter set, and static computation graph of the initial digital human inference model according to the target model format to obtain the target digital human inference model.

[0100] In this embodiment of the application, when the processor 30b parses the model file of the initial digital human inference model to obtain the network structure and weight parameter set of the initial digital human inference model, it is specifically used to: parse the sub-files with a specific format in the model file to obtain the weight parameter set of the initial digital human inference model; and define the model structure of the initial digital human inference model based on the forward propagation function in the model file to obtain the network structure of the initial digital human inference model.

[0101] In this embodiment of the application, when the processor 30b obtains a static computation graph of the initial digital human inference model by simulating the initial digital human inference model, it is specifically used to: input preset exemplary input data into the initial digital human inference model to simulate the operation of the initial digital human inference model; during the simulation operation, track and record the computational operations involved in each network layer of the initial digital human inference model and the dependencies between the computational operations; and construct a static computation graph based on the computational operations involved in each network layer and the dependencies between the computational operations.

[0102] In this embodiment of the application, the processor 30b performs control flow analysis on the initial digital human inference model to obtain a static computation graph of the initial digital human inference model. Specifically, it performs the following steps: compiling the forward propagation function in the model file of the initial digital human inference model to convert the control flow statements and computation operations in the forward propagation function into an initial computation graph; analyzing the dependencies between computation operations and adding the dependencies to the initial computation graph to obtain a static computation graph.

[0103] In this embodiment, when the processor 30b reorganizes the network structure, weight parameter set, and static computation graph of the initial digital human inference model according to the target model format to obtain the target digital human inference model, it specifically performs the following: based on the operator conversion table, it converts the operators in each network layer of the network structure into target operators under the target model format. The operators are used to implement the computational operations involved in the network layer. The operator conversion table stores the correspondence between the original operators under the original model format and the target operators under the target model format. Based on the weight parameter conversion relationship, it converts the original weight parameters of each network layer in the weight parameter set into target weight parameters under the target model format. Re-parameters; Weight parameter conversion relationship storage: the correspondence between the original weight parameters in the original model format and the target weight parameters in the target model format; Optimize network layers, data flow dependencies between network layers, and / or target operators in network layers according to the static computation graph to obtain optimized network layers, data flow dependencies between network layers, and / or target operators in network layers, as optimization objects; Generate the target digital human inference model based on the optimization objects, target weight parameters, and unoptimized objects. Unoptimized objects refer to the network layers, data flow dependencies between network layers, and target operators in network layers that have not been optimized.

[0104] In this embodiment of the application, when the processor 30b converts the operators in each network layer of the network structure into target operators in the target model format based on the operator conversion table, it specifically performs the following steps: traversing the original operators in each network layer of the network structure; for the currently traversed original operator, querying the operator conversion table; if the currently traversed original operator is found in the operator conversion table, replacing the currently traversed original operator with its corresponding target operator; if the currently traversed original operator is not found in the operator conversion table, replacing the currently traversed original operator with a custom operator, wherein the custom operator supports the target model format.

[0105] In this embodiment, the initial digital human inference model is a neural network model used to generate multiple facial region images with continuously changing mouth shapes based on audio feature information. The processor 30b is also used to acquire sample audio data, multiple labeled facial region images with continuously changing mouth shapes corresponding to the sample audio data, and sample frame rate; convert the sample audio data into sample audio feature information required by the basic digital human inference model, and perform segmentation operation on the sample audio feature information according to the duration and sample frame rate of the sample audio data to obtain multiple sample audio feature segments; input the multiple sample audio feature segments into the basic digital human inference model, map the multiple sample audio feature segments into multiple sample mouth deformation features, and generate multiple sample facial region images with continuously changing mouth shapes based on the multiple sample mouth deformation features; calculate the loss function between the multiple sample facial region images and the multiple labeled facial region images, and adjust the model parameters of the basic digital human inference model until the loss function reaches the set model training termination condition if the loss function does not reach the set model training termination condition, so as to obtain the initial digital human inference model.

[0106] Furthermore, such as Figure 3 As shown, the server also includes other components such as a communication component 30c, a display 30d, a power supply component 30e, and an audio component 30f. Figure 3 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 3 The components shown.

[0107] The detailed implementation methods and beneficial effects of the electronic devices provided in this application have been described in detail in the foregoing embodiments, and will not be elaborated further here.

[0108] Exemplary embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps in the above-described method embodiments.

[0109] An exemplary embodiment of this application also provides a computer program product comprising a computer program / instructions that, when executed by a processor, enable the processor to perform the steps described in the above method embodiments.

[0110] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0111] The aforementioned communication components are configured to facilitate wired or wireless communication between the device containing the communication components and other devices. The device containing the communication components can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication components receive broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication components also include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0112] The aforementioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0113] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0114] The aforementioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0115] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) containing computer-usable program code.

[0116] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0117] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0118] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0119] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interfaces, network interfaces, and memory.

[0120] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0121] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0122] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0123] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A model conversion method characterized by, Applied to a service terminal, the method comprises: receiving a model conversion request message sent by a mobile terminal, the model conversion request message comprising at least a system type of the mobile terminal, available resource information and a required model function of the mobile terminal; determining an initial digital human inference model adapted to the system type of the mobile terminal and the required model function from digital human inference models developed based on a target model development framework, the target model development framework being a model development framework applicable to the service terminal; in the case that the available resource information of the mobile terminal meets resource loading conditions required by a data packet, determining a target format conversion tool according to an original model format supported by the initial digital human inference model and a model format conversion tool mapping table, the model format conversion tool mapping table storing format conversion tools corresponding to digital human inference models supporting different model formats; calling the target format conversion tool to perform model format conversion processing on the initial digital human inference model within the same model framework, and generating a target digital human inference model supporting a target model format according to an optimization object, a target weight parameter and an unoptimized object in the model format conversion processing process, the target digital human inference model running on the mobile terminal to perform an inference task to generate a digital human video image with continuously changing mouth shape based on audio feature information; wherein the optimization object is an optimized network layer, data flow dependency relationship between network layers and / or target operator in the network layer obtained by performing a local network structure merging optimization operation on a network structure, data flow dependency relationship between network layers and / or target operator in the network layer and / or target operator in the network layer in the network structure and a folding optimization operation on a calculation constant involved in an inference process according to a static computation graph; the unoptimized object is an object in the network structure that has not been optimized; and the target weight parameter is a weight parameter of each network layer in the target model format.

2. The method of claim 1, wherein, The method further comprises: selecting a first candidate digital human inference model adapted to the system type of the mobile terminal from digital human inference models developed based on the target model development framework, different digital human inference models being configured with a list of operating systems to which they are adapted; determining a second candidate digital human inference model adapted to the required model function of the mobile terminal from the first candidate digital human inference model according to function description information of the first candidate digital human inference model; if the second candidate digital human inference model is one, taking the second candidate digital human inference model as the initial digital human inference model; and if the second candidate digital human inference model is multiple, selecting the one with the smallest model file as the initial digital human inference model.

3. The method of claim 1, wherein, The method further comprises: obtaining a model file of the initial digital human inference model. According to the size of the model file, it is judged whether the bandwidth information and / or the memory information in the available resource information of the mobile terminal meets the minimum bandwidth value and / or the minimum memory amount required in the resource loading condition; If yes, it is determined that the available resource information of the mobile terminal meets the resource loading condition.

4. The method of claim 1, wherein, The target format conversion tool is called to perform model format conversion processing on the initial digital human reasoning model, and during the model format conversion processing, a target digital human reasoning model supporting the target model format is generated according to the optimization object, the target weight parameter and the unoptimized object, including: The target format conversion tool is called to analyze the model file of the initial digital human reasoning model to obtain the network structure and the weight parameter set of the initial digital human reasoning model, the network structure being used to describe a plurality of network layers included in the initial digital human reasoning model and a connection relationship between the plurality of network layers, and the weight parameter set including original weight parameters of each network layer; And, the initial digital human reasoning model is simulated to run or control flow analysis is performed on the initial digital human reasoning model to obtain a static computation graph of the initial digital human reasoning model, the static computation graph including processing logic of each network layer from input data to output data and a data flow dependency relationship between network layers; According to the target model format, a local network structure merging optimization operation is performed on the network structure, the weight parameter set and the static computation graph of the initial digital human reasoning model, and a pre-located folding optimization operation is performed on a calculation constant involved in a reasoning process to obtain the target digital human reasoning model.

5. The method of claim 4, wherein, The model file of the initial digital human reasoning model is analyzed to obtain the network structure and the weight parameter set of the initial digital human reasoning model, including: The sub-file with a specific format in the model file is analyzed to obtain the weight parameter set of the initial digital human reasoning model; The model structure of the initial digital human reasoning model is defined based on the forward propagation function in the model file to obtain the network structure of the initial digital human reasoning model.

6. The method of claim 4, wherein, The initial digital human reasoning model is simulated to run to obtain the static computation graph of the initial digital human reasoning model, including: The preset exemplary input data is input into the initial digital human reasoning model to simulate the initial digital human reasoning model to run; During the simulation running, the calculation operations involved in each network layer of the initial digital human reasoning model and the dependency relationship between the calculation operations are tracked and recorded; According to the calculation operations involved in each network layer and the dependency relationship between the calculation operations, the static computation graph is constructed.

7. The method of claim 4, wherein, The initial digital human reasoning model is analyzed by control flow to obtain the static computation graph of the initial digital human reasoning model, including: The control flow statements and the calculation operations in the forward propagation function are converted into an initial computation graph by compiling the forward propagation function in the model file of the initial digital human reasoning model; The dependency relationship between the calculation operations is analyzed, and the dependency relationship is added to the initial computation graph to obtain the static computation graph.

8. The method of claim 4, wherein, The local network structure merging optimization operation of the network structure, the weight parameter set and the static computation graph of the initial digital human inference model and the pre-optimization operation of the calculation constant involved in the inference process are performed according to the target model format to obtain the target digital human inference model, including: The operators in each network layer in the network structure are converted into target operators in the target model format based on an operator conversion table, and the operators are used to implement the calculation operation involved in the network layer; the operator conversion table stores the corresponding relationship between the original operators in the original model format and the target operators in the target model format; The original weight parameters of each network layer in the weight parameter set are converted into target weight parameters in the target model format based on a weight parameter conversion relationship; the weight parameter conversion relationship stores the corresponding relationship between the original weight parameters in the original model format and the target weight parameters in the target model format; The local network structure merging optimization operation of the network layer, the data flow dependency relationship between the network layers and / or the target operator in the network layer in the network structure and the pre-optimization operation of the calculation constant involved in the inference process are performed according to the static computation graph to obtain the optimized network layer, the data flow dependency relationship between the network layers and / or the target operation in the network layer as an optimization object; The target digital human inference model is generated according to the optimization object, the target weight parameter and an unoptimized object, and the unoptimized object refers to the network layer, the data flow dependency relationship between the network layers and the target operator in the network layer in the network structure which are not optimized.

9. The method of claim 8, wherein, The operators in each network layer in the network structure are converted into target operators in the target model format based on an operator conversion table, including: The original operators in each network layer in the network structure are traversed; For the currently traversed original operator, the operator conversion table is queried; If the currently traversed original operator is queried in the operator conversion table, the currently traversed original operator is replaced with the target operator corresponding thereto; If the currently traversed original operator is not queried in the operator conversion table, the currently traversed original operator is replaced with a custom operator, and the custom operator supports the target model format.

10. The method according to any one of claims 1 to 9, characterized in that, The initial digital human inference model is a neural network model for generating a plurality of face region images with continuous changes in mouth shape based on audio feature information; the method further includes: Sample audio data, a plurality of labeled face region images with continuous changes in mouth shape corresponding to the sample audio data and a sample frame rate are obtained; The sample audio data is converted into sample audio feature information required by the basic digital human inference model, and the sample audio feature information is segmented according to the time length of the sample audio data and the sample frame rate to obtain a plurality of sample audio feature segments; inputting the plurality of sample audio feature segments into the base digital human inference model, mapping the plurality of sample audio feature segments into a plurality of sample mouth deformation features, and generating a plurality of sample face region images with continuously changing mouth shapes based on the plurality of sample mouth deformation features; calculating a loss function between the plurality of sample face region images and the plurality of labeled face region images, adjusting model parameters of the base digital human inference model until the loss function reaches a set model training termination condition, to obtain the initial digital human inference model, in a case where the loss function does not reach the set model training termination condition.

11. An electronic device, comprising: comprise: a memory and a processor; the memory is configured to store a computer program; and the processor is coupled to the memory and configured to execute the computer program to implement the steps in the method of any one of claims 1-10.

12. A computer readable storage medium storing a computer program, characterized in that, The computer program, when executed by the processor, causes the processor to implement the steps in the method of any one of claims 1-10.

13. A computer program product, characterised in that, The computer program product comprises computer programs / instructions, which, when executed by the processor, cause the processor to be able to implement the steps in the method of any one of claims 1-10.

Images