Model conversion method and device, storage medium and program product

Abstract

The embodiment of the invention provides a model conversion method and device, a storage medium and a program product. In the embodiment of the invention, under the condition that available resource information of a mobile terminal meets a resource loading condition required by a data packet, a target format conversion tool is determined according to an original model format supported by an initial digital human reasoning model and a model-format conversion tool mapping table; format conversion tools corresponding to the digital human reasoning models supporting different model formats are stored in the model-format conversion tool mapping table; and calling a target format conversion tool, and performing model format conversion processing on the initial digital human reasoning model to obtain a target digital human reasoning model which supports the target model format and is suitable for running in the mobile terminal, thereby improving the reasoning speed of the digital human model, and improving the reasoning efficiency of the digital human model. And moreover, the subsequent delay and lagging problems of the digital human video can be reduced, and the playing quality of the target digital human video is improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a model conversion method, device, storage medium and program product. Background Art

[0002] With the continuous development of artificial intelligence, AI digital human technology has also been widely used. At present, most solutions for generating digital humans rely on the server to complete the reasoning task and then transmit the generated AI digital human video to the mobile terminal.

[0003] However, this approach has the following problems: when the server needs to perform multiple reasoning tasks at the same time, the server may not respond in time due to heavy load, resulting in slow reasoning speed. In addition, after the server generates the digital human video, it needs to be transmitted to the mobile terminal through the network. If the network environment is poor, it may cause delays and freezes in playing the digital human video on the mobile terminal. Summary of the invention

[0004] Multiple aspects of the present application provide a model conversion method, device, storage medium and program product to improve the model reasoning speed and improve the playback quality of digital human videos.

[0005] An embodiment of the present application provides a model conversion method, which is applied to a service terminal. The method includes: receiving a model conversion request message sent by a mobile terminal, the model conversion request message at least including the system type of the mobile terminal, available resource information and the model function required by the mobile terminal; determining an initial digital human reasoning model adapted to the system type and the required model function of the mobile terminal from a digital human reasoning model developed based on a target model development framework, the target model development framework refers to a model development framework applicable to the server; when the available resource information of the mobile terminal meets the resource loading conditions required by the data packet, determining a target format conversion tool according to the original model format supported by the initial digital human reasoning model and a model-format conversion tool mapping table, the model-format conversion tool mapping table storing format conversion tools corresponding to digital human reasoning models supporting different model formats; calling the target format conversion tool to convert the model format of the initial digital human reasoning model to obtain a target digital human reasoning model supporting the target model format, the target digital human reasoning model being suitable for running on the mobile terminal.

[0006] The application embodiment also provides an electronic device, including: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the above-mentioned digital human live broadcast method.

[0007] The embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements the steps in the digital human live broadcast method.

[0008] The embodiment of the present application also provides a computer program product, which includes a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is enabled to implement the steps in the above-mentioned digital human live broadcast method.

[0009] In an embodiment of the present application, in a digital human reasoning model developed based on a target model development framework, an initial digital human reasoning model adapted to the system type of a mobile terminal and the required model functions is determined, and the target model development framework refers to a model development framework applicable to the server; when the available resource information of the mobile terminal meets the resource loading conditions required by the data packet, the target format conversion tool is determined according to the original model format supported by the initial digital human reasoning model and a model-format conversion tool mapping table, and the model-format conversion tool mapping table stores format conversion tools corresponding to digital human reasoning models supporting different model formats; the target format conversion tool is called to convert the model format of the initial digital human reasoning model to obtain a target digital human reasoning model that supports the target model format and is suitable for running on a mobile terminal, thereby improving the reasoning speed of the digital human model, and also reducing the subsequent delay and freeze problems of the digital human video, thereby improving the playback quality of the target digital human video. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0011] Figure 1 A flow chart of a model conversion method provided for an exemplary embodiment of the present application;

[0012] Figure 2 A schematic diagram of a process of generating a target digital human video provided by an exemplary embodiment of the present application;

[0013] Figure 3 A schematic structural diagram of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0015] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0016] In response to the existing technical problems, in an embodiment of the present application, in a digital human reasoning model developed based on a target model development framework, an initial digital human reasoning model adapted to the system type and required model functions of a mobile terminal is determined, and the target model development framework refers to a model development framework applicable to the server; when the available resource information of the mobile terminal meets the resource loading conditions required by the data packet, the target format conversion tool is determined according to the original model format supported by the initial digital human reasoning model and the model-format conversion tool mapping table, and the model-format conversion tool mapping table stores format conversion tools corresponding to digital human reasoning models supporting different model formats; the target format conversion tool is called to convert the model format of the initial digital human reasoning model to obtain a target digital human reasoning model that supports the target model format and is suitable for running on a mobile terminal, thereby improving the reasoning speed of the digital human model, and also reducing the subsequent delay and freeze problems of the digital human video, thereby improving the playback quality of the target digital human video.

[0017] A solution provided by an embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0018] Figure 1 The following is a flow chart of a model conversion method provided by an exemplary embodiment of the present application. Figure 1 As shown, the method is used on the server side, and the method includes:

[0019] 101. Receive a model conversion request message sent by a mobile terminal, where the model conversion request message at least includes a system type of the mobile terminal, available resource information, and a model function required by the mobile terminal;

[0020] 102. Determine an initial digital human reasoning model adapted to the system type and required model functions of the mobile terminal from the digital human reasoning model developed based on the target model development framework, where the target model development framework refers to a model development framework applicable to the server;

[0021] 103. When the available resource information of the mobile terminal meets the resource loading conditions required by the data packet, determine the target format conversion tool according to the original model format supported by the initial digital human reasoning model and the model-format conversion tool mapping table, wherein the model-format conversion tool mapping table stores format conversion tools corresponding to digital human reasoning models supporting different model formats;

[0022] 104. Calling a target format conversion tool to convert the initial digital human reasoning model into a model format, so as to obtain a target digital human reasoning model that supports the target model format, wherein the target digital human reasoning model is suitable for running on a mobile terminal.

[0023] In the embodiment of the present application, the system type includes but is not limited to: Android system, IOS system or Hongmeng system. Available resource information includes but is not limited to: CPU information, such as model and speed, number of cores and threads, and cache size; memory (RAM) information, such as total memory capacity, used and available memory, memory speed, and memory timing; storage information, such as hard disk or solid state drive capacity, available space, read and write speed, and interface type (such as SATA, NVMe); graphics processing unit (GPU) information, such as model, video memory size, and GPU usage; network information, such as network adapter type, IP address, network bandwidth, and network connection status; hardware status, such as device health status and fault information.

[0024] In an embodiment of the present application, an initial digital human reasoning model adapted to the system type and required model functions of a mobile terminal is determined from digital human reasoning models developed based on a target model development framework, including: selecting a first candidate digital human reasoning model adapted to the system type of the mobile terminal from digital human reasoning models developed based on a target model development framework, and different digital human reasoning models are configured with a list of operating systems adapted thereto; determining a second candidate digital human reasoning model adapted to the model functions required by the mobile terminal based on the functional description information of the first candidate digital human reasoning model; if there is one second candidate digital human reasoning model, using the second candidate digital human reasoning model as the initial digital human reasoning model; if there are multiple second candidate digital human reasoning models, selecting the one with the smallest model file as the initial digital human reasoning model.

[0025] Further optionally, a model file of the initial digital human reasoning model is obtained; based on the size of the model file, it is determined whether the bandwidth information and / or memory information in the available resource information of the mobile terminal meets the minimum bandwidth value and / or minimum memory amount required in the resource loading condition; if so, it is determined that the available resource information of the mobile terminal meets the resource loading condition.

[0026] In an embodiment of the present application, a target format conversion tool is called to convert the model format of the initial digital human reasoning model to obtain a target digital human reasoning model that supports the target model format, including: calling the target format conversion tool to parse the model file of the initial digital human reasoning model to obtain the network structure and weight parameter set of the initial digital human reasoning model, the network structure is used to describe the multiple network layers included in the initial digital human reasoning model and the connection relationship between the multiple network layers, the weight parameter set includes the original weight parameters of each network layer, and the weight parameters are used to indicate the degree of influence of the input data on the model output. Specifically, the weight determines the degree to which the input data should be amplified or reduced before being passed to the next layer; and, by simulating the initial digital human reasoning model or performing control flow analysis, a static calculation graph of the initial digital human reasoning model is obtained, the static calculation graph includes the processing logic of each network layer from input data to output data and the data flow dependency relationship between network layers; according to the target model format, the network structure, weight parameter set and static calculation graph of the initial digital human reasoning model are reorganized to obtain the target digital human reasoning model.

[0027] In an embodiment of the present application, the model file of the initial digital human reasoning model is parsed to obtain the network structure and weight parameter set of the initial digital human reasoning model, including: parsing a sub-file with a specific format in the model file to obtain the weight parameter set of the initial digital human reasoning model; defining the model structure of the initial digital human reasoning model based on the forward propagation function in the model file to obtain the network structure of the initial digital human reasoning model.

[0028] In an embodiment of the present application, a static computational graph of the initial digital human reasoning model is obtained by simulating the operation of the initial digital human reasoning model, including: inputting preset exemplary input data into the initial digital human reasoning model to simulate the operation of the initial digital human reasoning model; during the simulation operation, tracking and recording the computing operations involved in each network layer in the initial digital human reasoning model and the dependencies between the computing operations; and constructing a static computational graph based on the computing operations involved in each network layer and the dependencies between the computing operations.

[0029] In an embodiment of the present application, a control flow analysis is performed on the initial digital human reasoning model to obtain a static calculation graph of the initial digital human reasoning model, including: compiling the forward propagation function in the model file of the initial digital human reasoning model, converting the control flow statements and calculation operations in the forward propagation function into the initial calculation graph; analyzing the dependency relationship between the calculation operations, and adding the dependency relationship to the initial calculation graph to obtain a static calculation graph. Among them, the forward propagation function refers to.

[0030] In an embodiment of the present application, the network structure, weight parameter set and static calculation graph of the initial digital human reasoning model are reorganized according to the target model format to obtain the target digital human reasoning model, including: based on the operator conversion table, the operators in each network layer in the network structure are converted into the target operators in the target model format, and the operators are used to implement the calculation operations involved in the network layer; the operator conversion table stores the correspondence between the original operators in the original model format and the target operators in the target model format; based on the weight parameter conversion relationship, the original weight parameters of each network layer in the weight parameter set are converted into the target weight parameters in the target model format; the weight The heavy parameter conversion relationship stores the correspondence between the original weight parameters in the original model format and the target weight parameters in the target model format; according to the static calculation graph, the network layers in the network structure, the data flow dependencies between the network layers and / or the target operators in the network layers are optimized to obtain the optimized network layers, the data flow dependencies between the network layers and / or the target operations in the network layers as the optimization objects; according to the optimization objects, the target weight parameters and the unoptimized objects, the target digital human reasoning model is generated, and the unoptimized objects refer to the network layers in the network structure, the data flow dependencies between the network layers and the unoptimized objects in the target operators in the network layers.

[0031] In an embodiment of the present application, based on the operator conversion table, the operators in each network layer in the network structure are converted into the target operators under the target model format, including: traversing the original operators in each network layer in the network structure; for the original operator currently traversed, querying the operator conversion table; if the original operator currently traversed is queried in the operator conversion table, replacing the original operator currently traversed with the target operator corresponding to it; if the original operator currently traversed is not queried in the operator conversion table, replacing the original operator currently traversed with a custom operator, and the custom operator supports the target model format.

[0032] In an embodiment of the present application, the initial digital human reasoning model is a neural network model for generating multiple facial area images with continuously changing mouth shapes based on audio feature information; the method also includes: obtaining sample audio data, multiple labeled facial area images with continuously changing mouth shapes corresponding to the sample audio data, and a sample frame rate; converting the sample audio data into sample audio feature information required by the basic digital human reasoning model, and segmenting the sample audio feature information according to the duration and sample frame rate of the sample audio data to obtain multiple sample audio feature fragments; inputting the multiple sample audio feature fragments into the basic digital human reasoning model, mapping the multiple sample audio feature fragments into multiple sample mouth deformation features, and generating multiple sample facial area images with continuously changing mouth shapes based on the multiple sample mouth deformation features; calculating the loss function between the multiple sample facial area images and the multiple labeled facial area images, and if the loss function does not reach the set model training termination condition, adjusting the model parameters of the basic digital human reasoning model until the loss function reaches the set model training termination condition to obtain the initial digital human reasoning model.

[0033] To facilitate understanding, the conversion process of the initial digital human reasoning model is described in detail below with reference to specific examples.

[0034] The conversion process of the digital human reasoning model mainly includes the following steps: model input and analysis: extract the reasoning logic by loading weights and defining the model structure; calculation graph extraction: convert the dynamic graph of PyTorch into a static calculation graph, which is completed through tracing or scripting; operator mapping and weight migration: map the operators of PyTorch to the target framework and convert the weights; calculation graph optimization: optimize the calculation graph to improve the reasoning efficiency; generate the target format model: generate the final model file in the target format such as ONNX, TensorFlow; verification and adjustment: verify whether the converted model is consistent with the original model, and make necessary adjustments. The more specific process is as follows:

[0035] 1. Model input and parsing: When the inference model (such as PyTorch's .pth file) is input into the conversion tool, the first step is to parse and read the model's network structure and weight parameters. In PyTorch, this means loading the model and loading the saved weights using a predefined architecture. Specifically: Use torch.load() to load the .pth file, which will load the model's weight parameters. Next, the computational graph of the model is constructed through a predefined network architecture (usually implemented through the subclass torch.nn.Module). At this stage, the model's reasoning logic (that is, the model's forward propagation forward() function) is extracted, recording the operations of each layer of the model and its data flow.

[0036] 2. Extraction of model computation graph (Tracing or Scripting): In order to convert the model to the target format, the model's computation graph must be converted from a dynamic graph framework (such as PyTorch) to a static graph. This can be done in two ways:

[0037] A. Tracing mode

[0038] In Tracing mode, the conversion tool will track the sequence of operations of the model during forward propagation by feeding the model a set of example inputs. This method can capture the computational graph of the model, but if the model has branching logic or dynamic operations, tracing may not be able to fully capture these dynamic behaviors. For example, control flow operations (such as if-else statements) may be ignored. Specifically, the conversion tool will pass example inputs to the model, trace all tensor operations involved; record the computational operations of each layer (such as convolution, full connection, pooling, etc.) and the corresponding weights; and establish a complete computational graph that describes how to transform from input data through each layer to output data.

[0039] B. Scripting mode

[0040] In Scripting mode, the Python code of the model is converted into a static graph, which can handle more complex dynamic control flows. This method is more suitable for models with complex dynamic logic. The conversion tool analyzes each step of the forward() function and generates a static graph. Specifically, the conversion tool compiles the forward() function of the model, converts the control flow (such as if, for statements) and tensor operations into a static computational graph, analyzes the dependencies of each operation, and generates a complete computational graph for the model.

[0041] 3. Conversion between operators and weights

[0042] The generated computational graph contains the hierarchical structure of the model, the type of operation at each layer (such as convolution, pooling, activation function, etc.), and the data flow dependencies between them. Next, the tool needs to convert these operators and weights into the target format.

[0043] A. Operator Mapping

[0044] Different deep learning frameworks (such as PyTorch, TensorFlow, ONNX) have their own sets of operators. During the conversion process, the tool maps PyTorch's operators to the operators of the target framework according to the requirements of the target format. For example: PyTorch's torch.nn.Conv2d will be mapped to the Conv operator in ONNX; PyTorch's ReLU activation function will be mapped to the Relu operator in ONNX. This mapping is achieved through an operator conversion table. The tool checks the operation type of each layer and looks for the corresponding operator in the target format. If some operations are not supported in the target format, manual adjustments or custom operators may be required.

[0045] B. Weight Migration

[0046] While converting operators, the tool also needs to extract the weight parameters of each layer (such as convolution kernels, biases, etc.) from .pth and convert them to the target format. For example, weights in PyTorch are usually torch.Tensor objects, while in ONNX, weights will be converted to standard multidimensional array format; these weights need to be saved according to the format and requirements of the target framework; the weight migration process ensures that the model can use the same trained parameters for inference in the target framework.

[0047] 4. Optimization of computational graph

[0048] After generating the computational graph in the target format, the conversion tool usually performs a series of optimizations to improve the efficiency of model reasoning. Common optimizations include: Layer merging: For example, convolutional layers and batch normalization (BatchNorm) layers are merged into one operation to reduce the amount of calculation; Constant folding: For some static operations, such as addition and multiplication, the tool may calculate the results in advance during the compilation phase to reduce the computational burden during reasoning; Deleting redundant nodes: If the model contains unused layers or operations, the tool will delete them to optimize the size and speed of the model.

[0049] 5. Generate target format model

[0050] After completing the operator mapping and weight migration, the tool will generate the final model file according to the target format standard. For example, when converting to ONNX, the tool will generate a .onnx file containing the network structure and weights, which can be run in an ONNX-compatible reasoning engine (such as ONNX Runtime); when converting to TensorFlow, it may generate a .pb file or save it in the TensorFlow SavedModel format.

[0051] 6. Model verification and adjustment

[0052] After the conversion is completed, the last step is to verify the consistency of the model. The tool usually verifies whether the model has been successfully converted in the following ways: Run inference verification: Use the same input data to perform inference in the original model and the converted model to check whether the output is consistent; Adjust incompatible layers: If some layers are not fully supported in the target format, the tool may generate warnings and allow developers to make manual adjustments. For example, some special custom layers may need to be rewritten or provide alternative implementations.

[0053] For example, when converting a .pth model to .onnx format, the format conversion of the model's network structure and weights is mainly involved, but the core working principle and structure of the model remain largely unchanged. The following is a detailed description:

[0054] In PyTorch, a .pth file usually contains two parts: model weight parameters: these are all the parameters of the trained model, such as the weights and biases of each layer of the network; network structure: this is the code logic that defines how the model receives input, calculates through different layers, and produces output. When converted to .onnx, the network structure and weights of the model will be converted into the .onnx format. This process preserves the network's computational graph and trained weights. For the conversion of weight parameters, the model's weight parameters are extracted from .pth and saved in .onnx. These parameters will be stored in a format compatible with ONNX. For the conversion of network structure: the structure of the model will also be converted from PyTorch's computational graph representation to the ONNX format. This includes the types of layers (such as convolutional layers, fully connected layers, activation functions, etc.) and how they are connected.

[0055] It should be noted that during the conversion process, the network structure usually does not change, but there are some situations that require attention:

[0056] 1. Layers that are not fully compatible between PyTorch and ONNX: Some special layers or operations implemented in PyTorch may not be directly converted to ONNX format. These layers may be replaced with equivalent layers supported by ONNX. For example, some specific custom layers or operations, such as some uncommon dynamic operations, may require manual adjustments or custom implementations during conversion.

[0057] 2. The difference between static graphs and dynamic graphs: PyTorch is a deep learning framework based on dynamic graphs, which means that the network structure is dynamically constructed at runtime; while ONNX is a static graph, which means that the network structure needs to be fixed during conversion, and all dynamic operations need to be converted into static computational graphs. This conversion may affect some dynamic structures, resulting in the need to rewrite the network structure into a compatible static version.

[0058] 3. Changes in working principles: Usually the basic working principle of the model will not change, but if you encounter incompatible network layers during conversion (dynamic operations as described above), you need to make some adjustments to the model to ensure that it can be correctly inferred in the ONNX runtime (such as ONNX Runtime or other frameworks that support ONNX). 4. The main network layers of the converted model: Convolutional layers: Generally, they will not change and are still used for feature extraction. Fully connected layers: Consistent with PyTorch, they are still used for classification or regression. Activation functions: Most common activation functions (such as ReLU, Sigmoid) will be retained intact. If there is a network involving a dynamic structure, it may be implemented through some static alternatives of ONNX after conversion, and the working principle will be slightly adjusted.

[0059] 4. Notes on conversion: Model input and output: PyTorch may allow multiple forms of input, while ONNX requires explicit input and output tensor shapes. This may require you to fix the input size during the conversion process, or explicitly specify dynamic dimensions. Operations during export: PyTorch torch.onnx.export() usually needs to be called

[0060] method to convert, and when exporting, make sure to provide the correct example input so that the model's calculation graph can be exported correctly.

[0061] 5. It should also be noted that when converting the Wav2Lip .pth model to the .onnx format, the network structure and weight parameters are mainly converted, and the core network structure will not change. If the model contains layers or operations that are incompatible with PyTorch and ONNX, some static substitution adjustments may be made to these layers. In more special cases, manual adjustments may be required based on the specific model and operation to ensure that the network structure in ONNX is consistent with that in PyTorch.

[0062] The embodiment of the present application provides a model conversion process, in which an initial digital human reasoning model adapted to the system type and required model functions of a mobile terminal is determined in a digital human reasoning model developed based on a target model development framework, wherein the target model development framework refers to a model development framework applicable to a server; when the available resource information of the mobile terminal meets the resource loading conditions required by the data packet, a target format conversion tool is determined according to the original model format supported by the initial digital human reasoning model and a model-format conversion tool mapping table, wherein the model-format conversion tool mapping table stores format conversion tools corresponding to digital human reasoning models supporting different model formats; the target format conversion tool is called to convert the model format of the initial digital human reasoning model to obtain a target digital human reasoning model supporting the target model format and suitable for running on a mobile terminal, thereby improving the reasoning speed of the digital human model, and also reducing the delay and freeze problems of the subsequent digital human video, thereby improving the playback quality of the target digital human video.

[0063] Combine the following Figure 2 The generation process of the target digital human video is explained in detail.

[0064] S1. Acquire audio data for generating digital human voice, an image resource package for generating digital human image, and a frame rate of a digital human video to be generated, wherein the image resource package includes a plurality of original digital human base maps, an original face region image in each digital human base map, and position information of each original face region image in the corresponding original digital human base map;

[0065] S2, converting the audio data into target audio feature information required by the target digital human reasoning model, and segmenting the target audio feature information according to the duration and frame rate of the audio data to obtain multiple audio feature segments;

[0066] S3, inputting multiple audio feature segments into the target digital human inference model, mapping the multiple audio feature segments into multiple mouth deformation features, and generating multiple target facial region images with continuously changing mouth shapes based on the multiple mouth deformation features;

[0067] S4, according to the position information of each original face region image in the corresponding original digital human base image, replacing the original face region images in the multiple original digital human base images with the multiple target face region images, so as to obtain multiple target digital human images with continuously changing mouth shapes;

[0068] S5. Generate a digital human action picture based on multiple target digital human images with continuously changing mouth shapes, and drive the digital human mouth in the digital human action picture to broadcast audio data to obtain the target digital human video.

[0069] In the embodiments of the present application, there is no limitation on the implementation form of the mobile terminal. The mobile terminal may be a smart handheld device, such as a smart phone, a tablet computer, a laptop computer or a desktop computer, etc.; or, the mobile terminal may also be a smart wearable device, such as a smart watch, a smart bracelet, etc.; or, the mobile terminal may also be various smart home appliances with display screens, such as a smart TV, a smart large screen or a smart robot, etc.

[0070] In an embodiment of the present application, audio data refers to audio data of voice content contained in the audio, which may be recorded in advance, or may be audio data generated in advance or in real time by a server or mobile terminal based on text information. The timing of generating audio data depends on the specific application scenario, and this embodiment does not specifically limit this. For example, the real-time and flexible conversation audio data in an AI interview room may be audio data for answering generated in real time based on text information corresponding to the input content of the dialogue party, and the audio data corresponding to the fixed words in some dialogue scenarios may be recorded in advance or generated in advance based on text information.

[0071] In order to determine the deformation of the mouth with audio data, this embodiment introduces a digital human reasoning model. The digital human reasoning model is at least an AI model for inferring the mouth shape based on audio data. Specifically, it can predict and generate lip shape animations synchronized with the audio data based on the input audio data, such as being used to generate realistic animations of digital humans or virtual characters, so that the characters can move their mouths naturally according to the audio data, thereby improving the accuracy of identifying mouth deformation. The embodiment of the present application does not limit the type of digital human reasoning model. For example, the digital human reasoning model can be Wav2Lip, ER-NeRF, VideoRetalking, MuseTalk, EchoMimic, and the like. Among them, Wav2Lip is an open source lip synchronization tool that can generate lip animations that match the input voice. It is suitable for any face and any language, and can generate realistic lip movements; ER-NeRF is a real-time responsive Talking Portrait Synthesis model that can synchronize the given audio with the mouth shape of the characters in the video. It should be noted that specific videos need to be trained to generate the corresponding digital human reasoning model;

[0072] VideoRetalking can synchronize the mouth shape of the video character with the input sound. The model includes face video generation, audio-driven lip synchronization and facial enhancement; MuseTalk is a real-time audio-driven lip synchronization model that supports multiple language inputs and is particularly suitable for high-definition facial areas; EchoMimic can give static images vivid voice and expressions. It combines audio and facial landmarks through a deep learning model to create highly realistic dynamic portrait videos. These models analyze the input voice signal and generate corresponding mouth animations, so that the mouth shape of the digital human character when speaking matches the actual voice content. While improving the accuracy of recognizing mouth deformation, it can also improve the realism of the character and the audience's immersion. For the sake of distinction and description, the digital human reasoning model in actual application is called the target digital human reasoning model. The focus of this embodiment is the Wav2Lip model.

[0073] It should be noted that the target digital human reasoning model is obtained by converting the model format of the initial digital human reasoning model, and the initial digital human reasoning model is developed on the server side through a target model development framework suitable for the server side. Specifically, when the mobile terminal has a demand for the target digital human model, a model conversion request message is sent to the server side, and the model conversion request message includes the system type of the mobile terminal, available resource information, and model functions required by the mobile terminal, so as to request the server side to convert the initial digital human reasoning model into the target digital human reasoning model based on the system type, available resource information, and model functions; further, a model conversion response message is received from the server side, and the model conversion response message is used to notify the mobile terminal to obtain the model file of the target digital human reasoning model from the remote persistent storage space; the model file of the target digital human reasoning model is obtained from the remote persistent storage space and stored in the local persistent storage space, and when the target digital human reasoning model is used, the model file is loaded from the local persistent storage space to run the target digital human reasoning model.

[0074] It should also be noted that the development process of the initial digital human model may include the training process of the initial digital human model, or the development process of the initial digital human model may not include the training process of the initial digital human model. In the case where the development process of the initial digital human model does not include the training process of the initial digital human model, the training process of the initial digital human model may be completed on the server side before the model format conversion, or on the mobile terminal before the model format conversion, or on the mobile terminal after the model format conversion. The development, training and model format conversion process of the initial digital human reasoning model can be referred to the relevant description of the following embodiments, which will not be repeated here.

[0075] In the embodiment of the present application, the specific implementation method of the target digital human reasoning model running on the mobile terminal is not limited. For example, the mobile terminal can directly run the target digital human reasoning model; or, the target digital human reasoning model runs on an application on the mobile terminal. For the convenience of description and distinction, the application is referred to as the target application. In addition, this embodiment does not limit the implementation form of the target application. The target application can be an independently running APP, or a web page or a small program that depends on the APP to run.

[0076] In an embodiment of the present application, in order to obtain a digital human video, audio data for generating the digital human voice and an image resource package for generating the digital human image may be first obtained, so as to infer the mouth shape changes corresponding to the digital human image based on the audio data, and further obtain the digital human video based on the audio data and the mouth shape changes. Among them, the image resource package may only contain multiple original digital human base maps, or may simultaneously contain multiple original digital human base maps, original facial area images in each digital human base map, and position information of each original facial area image in the corresponding original digital human base map, which is not limited in this embodiment. Multiple original digital human base maps may include multiple groups of original digital human base maps, and each group of original digital human base maps may be multiple continuous digital human images captured from any digital human video, and the mouth shapes in the multiple continuous digital human images are continuously changing.

[0077] The original data format of the audio data may be different from the data format required by the target digital human reasoning model. After obtaining the audio data, in order to facilitate the target digital human reasoning model to infer the mouth shape changes corresponding to the audio data, the original data format of the audio data can be converted to the data format required by the target digital human reasoning model, that is, the initial audio feature information of the audio data is converted to the target audio feature information required by the target digital human reasoning model. It should be noted that the types of audio feature information required by different digital human reasoning models may be the same or different. Taking the Wav2Lip model as an example, the original data format of the audio data is .wav format, and the data format required by the Wav2Lip model is the Mel spectrum data format. Then, the audio data needs to be converted from .wav format to Mel spectrum data format, that is, the initial audio feature information of the audio data is converted to the Mel spectrum data required by the Wav2Lip model. The Mel spectrum data corresponds to a Mel spectrum graph.

[0078] In an embodiment of the present application, audio data is converted into audio feature information required by a target digital human reasoning model, including: extracting Fourier transform data from the audio data; generating Mel spectrum data based on the Fourier transform data; and cropping, padding or resizing the Mel spectrum data according to the requirements of the target digital human reasoning model for input data to obtain target audio feature information. Among them, cropping refers to removing edge noise in the Mel spectrum data or retaining the required spectrum portion. Padding refers to adding zero values ​​or other specific values ​​to the edges of the data. Usually, when the size of the Mel spectrum data does not meet the input requirements of the model, the size of the data can be adjusted by padding to meet the requirements. Sizing refers to resampling or interpolation to change the resolution or size of the Mel spectrum data to adapt it to a specific input format. The target audio feature information refers to the Mel spectrum feature information corresponding to the Mel spectrum data processed based on the requirements of the input data of the target digital human reasoning model.

[0079] In an optional embodiment, Fourier transform data is extracted from audio data, and spectrum data is generated according to the Fourier transform data, including: first, the audio data needs to be loaded using an appropriate library, such as using the Librosa library to load audio data in .wav format; preprocessing the audio data, the preprocessing includes but is not limited to normalizing the volume, removing noise and other steps to improve the accuracy of subsequent processing; converting the audio data into an audio signal, the audio signal corresponding to key feature information; performing a short-time Fourier transform (Fast Fourier Transform, FFT) on the audio signal to convert the audio signal from the time domain to the frequency domain; using a Mel filter to convert the audio signal in the frequency domain to the Mel scale to generate a Mel spectrum map, the Mel spectrum map corresponding to the Mel spectrum data. Further, the Mel spectrum map can also be subjected to a discrete cosine transform (Discrete Cosine Transform, DCT) in English, which is the full name of extracting Mel frequency cepstral coefficients (Mel Frequency Cepstral Coefficients, MFCCs), the Mel frequency cepstral coefficient is a feature representation of the Mel spectrum data, that is, the Mel frequency cepstral coefficient is the Mel spectrum feature information corresponding to the Mel spectrum data.

[0080] Further optionally, in order to improve the accuracy of the target digital human reasoning model's reasoning, the target audio feature information can be segmented, and the target digital human reasoning model can be used to reason about each segment of audio feature information separately to obtain multiple digital human images with continuously changing mouth shapes.

[0081] In an optional embodiment, the target feature information can be segmented according to the duration of the audio data and the frame rate of the digital human video to be generated. The frame rate refers to the number of static images (frames) that can be displayed per second during video playback. The frame rate is an important indicator of the smoothness of video playback, which is usually measured in "frames per second" (FPS). For example, if the frame rate of the digital human to be generated is 25fps, 25 pictures will be played and explained within 1 second. Based on this, the target audio feature information is segmented, including: obtaining the frame rate of the digital human video to be generated; according to the duration of the audio data and the frame rate of the digital human video to be generated, the target feature information is segmented to obtain multiple audio feature fragments.

[0082] Among them, when obtaining the frame rate of the digital human video to be generated, it is necessary to consider the purpose of the digital human video, the audience's viewing device, the video playback platform, and the required visual effects. For example, if the digital human video is used for online playback and needs to show smooth facial expressions and synchronized lip movements, a frame rate of 30fps or higher can be selected; if the digital human video is used in a movie or art project, 24fps can be selected to meet the viewing experience. In addition, technical limitations and costs need to be considered. Usually, a higher frame rate means that more data needs to be processed during editing and rendering, which may increase the demand and cost of computing resources. At the same time, the size of the high-frame-rate digital human video file will also increase, which may affect storage and transmission efficiency. In practical applications, if applied to cloud intelligent digital human services, the video frame rate is generally not less than 25fps and not higher than 60fps. If applied to silicon-based digital human image cloning, the video frame rate is generally 30fps or 60fps, so that the smoothness and detail of the video can be ensured while taking into account the technical and cost limitations in practical applications.

[0083] Optionally, the target audio feature information is segmented according to the duration and frame rate of the audio data to obtain multiple audio feature segments, including: taking the duration of the audio data as the dividend and the frame rate as the divisor, performing a quotient operation to obtain the video frame duration, which means the duration of each frame of the video; taking the video frame duration as the segmentation unit, the target audio feature information is segmented to obtain multiple audio feature segments. For example, if the duration of the audio data is 50s and the frame rate of the digital human video to be generated is 25fps, then

[0084] Furthermore, after obtaining multiple audio feature fragments, the multiple audio feature fragments can be input into the target digital human inference model, the multiple audio feature fragments can be mapped into multiple mouth deformation features, and multiple target facial area images with continuously changing mouth shapes can be generated based on the multiple mouth deformation features.

[0085] In an embodiment of the present application, multiple audio feature fragments are input into a target digital human inference model, the multiple audio feature fragments are mapped into multiple mouth deformation features, and multiple target facial area images with continuously changing mouth shapes are generated based on the multiple mouth deformation features, including: inputting multiple audio feature fragments into a target digital human inference model, inferring the changes in the digital human's mouth shape based on the multiple audio feature fragments, and obtaining multiple mouth deformation features; converting the multiple mouth deformation features into multiple pixel point data in the facial area and position distribution information of each pixel point; generating multiple target facial area images according to the multiple pixel point data in the facial area corresponding to the multiple mouth deformation features and the position distribution information of each pixel point.

[0086] In an optional embodiment, the target digital human inference model includes: an audio feature processing network layer, which infers the changes in the digital human's mouth shape based on multiple audio feature segments to obtain multiple mouth deformation features, including: inputting multiple audio feature segments into the audio feature processing network layer to generate mouth deformation features corresponding to each audio feature segment. Among them, each mouth deformation feature is a multidimensional feature vector, and a multidimensional feature vector at least includes a mouth shape parameter, an opening and closing degree parameter, and a lip position parameter. The mouth shape parameter is used to represent the current mouth shape, the opening and closing degree parameter is used to represent the current mouth opening and closing degree, and the lip position parameter is used to represent the current mouth position in the facial area.

[0087] Further optionally, the target digital human inference model may also include: a visual feature processing network layer, which converts multiple mouth deformation features into multiple pixel point data in the facial area and position distribution information of each pixel point, including: inputting multiple mouth deformation features into the video feature processing network layer, and based on the mouth shape parameters, opening and closing parameters and lip position parameters corresponding to each mouth deformation feature, converting each mouth deformation feature into multiple pixel point data in the facial area and position distribution information of each pixel point.

[0088] Further optionally, the target digital human inference model also includes a generator. After obtaining multiple pixel point data and position distribution information of each pixel point, the multiple pixel point data and the position distribution information of each pixel point can be input into the generator, and based on the multiple pixel point data and the position distribution information of each pixel point, multiple target facial area images with continuously changing mouth shapes can be obtained.

[0089] It can be seen from the above embodiments that the image resource package includes multiple original digital human base maps, original facial area images in each digital human base map, and position information of each original facial area image in the corresponding original digital human base map. After obtaining multiple target facial area images with continuously changing mouth shapes, the original facial area images in the multiple original digital human base maps can be replaced with the multiple target facial area images according to the position information of each original facial area image in the corresponding original digital human base map to obtain multiple target digital human images with continuously changing mouth shapes.

[0090] Furthermore, after obtaining a plurality of target digital human images with continuously changing mouth shapes, a target digital human video can be obtained based on the plurality of target digital human images with continuously changing mouth shapes. This embodiment does not limit the specific implementation method of obtaining the target digital human video based on the plurality of target digital human images with continuously changing mouth shapes, and different implementation methods may correspond to different application scenarios.

[0091] In an optional embodiment, a target digital human video is obtained based on a plurality of target digital human images with continuously changing mouth shapes, including: generating a digital human action picture based on the plurality of target digital human images with continuously changing mouth shapes, and driving the digital human mouth in the digital human action picture to broadcast the audio data to obtain the target digital human video. The digital human action picture and the broadcast audio in the target digital human video are two independent parts, which is suitable for real-time dialogue scenarios, such as real-time question and answer in AI live broadcast rooms / interview rooms.

[0092] In another optional embodiment, a target digital human video is obtained according to a plurality of target digital human images with continuously changing mouth shapes, including: generating a target digital human video according to the plurality of target digital human images with continuously changing mouth shapes and audio data for generating digital human voices. The target digital human video is a unified whole with the action picture of the digital human and the spoken audio, which is suitable for non-real-time playback scenarios, such as non-real-time live broadcast digital human live video. In this embodiment, correspondingly, the generator in the target digital human reasoning model can also obtain the target digital human video according to the plurality of target digital human images with continuously changing mouth shapes. Further, the target digital human reasoning model can also include a lip synchronization discriminator for evaluating the synchronization accuracy between the generated video frame and the audio data. In the case where the evaluation result does not meet the synchronization accuracy threshold, the target digital human reasoning model can be retrained based on the synchronization accuracy standard, and the target digital human video is generated using the retrained target digital human reasoning model based on the plurality of target digital human images with continuously changing mouth shapes and the audio data for generating digital human voices. Furthermore, the target digital human reasoning model may also include a visual quality discriminator for evaluating whether the generated video frames look natural visually and have no obvious generation traces. Similarly, if the evaluation result does not meet the synchronization accuracy threshold, the target digital human reasoning model can be retrained based on the synchronization accuracy standard, and the retrained target digital human reasoning model can be used to generate the target digital human video based on multiple target digital human images with continuously changing mouth shapes and audio data for generating digital human voices.

[0093] In the above embodiments of the present application, the original reasoning environment of the target digital human reasoning model is changed from reasoning operation on the server to reasoning on the mobile terminal, which can avoid the problem that when the server needs to perform multiple reasoning tasks at the same time, the server may not respond in time due to excessive load, resulting in slow reasoning speed, thereby improving the reasoning speed of the target digital human model. In addition, the target digital human reasoning model is used to infer multiple target facial area images with continuously changing mouth shapes based on multiple audio feature segments corresponding to audio data, and a digital human action picture is generated based on multiple target digital human images with continuously changing mouth shapes, and the digital human in the digital human action picture is driven to broadcast audio data to obtain the target digital human video, which can not only improve the adaptability of the mouth shape and the audio content, but also reduce the delay and freeze problems in playing the digital human video, thereby improving the playback quality of the target digital human video.

[0094] In the generation process of the digital human video provided in the application embodiment, the initial digital human reasoning model is a neural network model for generating multiple facial area images with continuously changing mouth shapes based on audio feature information; the method also includes: obtaining sample audio data, multiple labeled facial area images with continuously changing mouth shapes corresponding to the sample audio data, and sample frame rate; converting the sample audio data into sample audio feature information required by the basic digital human reasoning model, and segmenting the sample audio feature information according to the duration and sample frame rate of the sample audio data to obtain multiple sample audio feature fragments; inputting the multiple sample audio feature fragments into the basic digital human reasoning model, mapping the multiple sample audio feature fragments into multiple sample mouth deformation features, and generating multiple sample facial area images with continuously changing mouth shapes based on the multiple sample mouth deformation features; calculating the loss function between the multiple sample facial area images and the multiple labeled facial area images, and when the loss function does not reach the set model training termination condition, adjusting the model parameters of the basic digital human reasoning model until the loss function reaches the set model training termination condition to obtain the initial digital human reasoning model.

[0095] Figure 3 The following is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. Figure 3 As shown, it includes: a memory 30a and a processor 30b; the memory 30a is used to store a computer program; the processor 30b is coupled to the memory 30a and is used to execute the computer program to implement the following steps:

[0096] Receive a model conversion request message sent by a mobile terminal, the model conversion request message at least including the system type of the mobile terminal, available resource information and the model functions required by the mobile terminal; determine an initial digital human reasoning model adapted to the system type and the required model functions of the mobile terminal from a digital human reasoning model developed based on a target model development framework, the target model development framework refers to a model development framework applicable to the server; when the available resource information of the mobile terminal meets the resource loading conditions required by the data packet, determine a target format conversion tool according to the original model format supported by the initial digital human reasoning model and a model-format conversion tool mapping table, the model-format conversion tool mapping table stores format conversion tools corresponding to digital human reasoning models supporting different model formats; call the target format conversion tool to convert the model format of the initial digital human reasoning model to obtain a target digital human reasoning model supporting the target model format, the target digital human reasoning model being suitable for running on the mobile terminal.

[0097] In an embodiment of the present application, when the processor 30b determines the initial digital human reasoning model that is adapted to the system type and required model functions of the mobile terminal from the digital human reasoning model developed based on the target model development framework, it is specifically used to: select a first candidate digital human reasoning model that is adapted to the system type of the mobile terminal from the digital human reasoning model developed based on the target model development framework, and different digital human reasoning models are configured with a list of operating systems that they are adapted to; based on the functional description information of the first candidate digital human reasoning model, determine the second candidate digital human reasoning model that is adapted to the model function required by the mobile terminal; if there is one second candidate digital human reasoning model, use the second candidate digital human reasoning model as the initial digital human reasoning model; if there are multiple second candidate digital human reasoning models, select the one with the smallest model file as the initial digital human reasoning model.

[0098] Further optionally, the processor 30b is also used to obtain a model file of the initial digital human reasoning model; based on the size of the model file, determine whether the bandwidth information and / or memory information in the available resource information of the mobile terminal meets the minimum bandwidth value and / or minimum memory amount required in the resource loading condition; if so, determine that the available resource information of the mobile terminal meets the resource loading condition.

[0099] In an embodiment of the present application, when the processor 30b calls the target format conversion tool to convert the model format of the initial digital human reasoning model to obtain a target digital human reasoning model that supports the target model format, it is specifically used to: call the target format conversion tool to parse the model file of the initial digital human reasoning model to obtain the network structure and weight parameter set of the initial digital human reasoning model, the network structure is used to describe the multiple network layers included in the initial digital human reasoning model and the connection relationship between the multiple network layers, and the weight parameter set includes the original weight parameters of each network layer; and, by simulating the initial digital human reasoning model or performing control flow analysis, a static calculation graph of the initial digital human reasoning model is obtained, the static calculation graph includes the processing logic of each network layer from input data to output data and the data flow dependency relationship between network layers; according to the target model format, the network structure, weight parameter set and static calculation graph of the initial digital human reasoning model are reorganized to obtain the target digital human reasoning model.

[0100] In an embodiment of the present application, when the processor 30b parses the model file of the initial digital human reasoning model to obtain the network structure and weight parameter set of the initial digital human reasoning model, it is specifically used to: parse the sub-file with a specific format in the model file to obtain the weight parameter set of the initial digital human reasoning model; define the model structure of the initial digital human reasoning model based on the forward propagation function in the model file to obtain the network structure of the initial digital human reasoning model.

[0101] In an embodiment of the present application, when the processor 30b simulates the initial digital human reasoning model to obtain a static calculation graph of the initial digital human reasoning model, it is specifically used to: input preset exemplary input data into the initial digital human reasoning model to simulate the operation of the initial digital human reasoning model; during the simulation operation, track and record the computing operations involved in each network layer in the initial digital human reasoning model and the dependencies between the computing operations; and construct a static calculation graph based on the computing operations involved in each network layer and the dependencies between the computing operations.

[0102] In an embodiment of the present application, the processor 30b performs control flow analysis on the initial digital human reasoning model to obtain a static computation graph of the initial digital human reasoning model, specifically for: compiling the forward propagation function in the model file of the initial digital human reasoning model, converting the control flow statements and computation operations in the forward propagation function into an initial computation graph; analyzing the dependencies between computation operations, and adding the dependencies to the initial computation graph to obtain a static computation graph.

[0103] In an embodiment of the present application, when the processor 30b reorganizes the network structure, weight parameter set and static calculation graph of the initial digital human reasoning model according to the target model format to obtain the target digital human reasoning model, it is specifically used to: based on the operator conversion table, convert the operators in each network layer in the network structure into the target operator in the target model format, and the operator is used to implement the calculation operations involved in the network layer; the operator conversion table stores the correspondence between the original operator in the original model format and the target operator in the target model format; based on the weight parameter conversion relationship, convert the original weight parameters of each network layer in the weight parameter set into the target weight parameter in the target model format; Re-parameters; the weight parameter conversion relationship stores the correspondence between the original weight parameters in the original model format and the target weight parameters in the target model format; according to the static calculation graph, the network layers in the network structure, the data flow dependencies between the network layers and / or the target operators in the network layers are optimized to obtain the optimized network layers, the data flow dependencies between the network layers and / or the target operations in the network layers as the optimization objects; according to the optimization objects, the target weight parameters and the unoptimized objects, the target digital human reasoning model is generated, and the unoptimized objects refer to the network layers in the network structure, the data flow dependencies between the network layers and the target operators in the network layers that have not been optimized.

[0104] In an embodiment of the present application, when the processor 30b converts the operators in each network layer in the network structure into the target operator under the target model format based on the operator conversion table, it is specifically used to: traverse the original operators in each network layer in the network structure; for the original operator currently traversed, query the operator conversion table; if the original operator currently traversed is found in the operator conversion table, replace the original operator currently traversed with the target operator corresponding to it; if the original operator currently traversed is not found in the operator conversion table, replace the original operator currently traversed with a custom operator, and the custom operator supports the target model format.

[0105] In an embodiment of the present application, the initial digital human reasoning model is a neural network model for generating multiple facial area images with continuously changing mouth shapes based on audio feature information; the processor 30b is also used to obtain sample audio data, multiple labeled facial area images with continuously changing mouth shapes corresponding to the sample audio data, and a sample frame rate; convert the sample audio data into sample audio feature information required by the basic digital human reasoning model, and segment the sample audio feature information according to the duration and sample frame rate of the sample audio data to obtain multiple sample audio feature fragments; input the multiple sample audio feature fragments into the basic digital human reasoning model, map the multiple sample audio feature fragments into multiple sample mouth deformation features, and generate multiple sample facial area images with continuously changing mouth shapes based on the multiple sample mouth deformation features; calculate the loss function between the multiple sample facial area images and the multiple labeled facial area images, and if the loss function does not reach the set model training termination condition, adjust the model parameters of the basic digital human reasoning model until the loss function reaches the set model training termination condition to obtain the initial digital human reasoning model.

[0106] Furthermore, if Figure 3 As shown, the server also includes: a communication component 30c, a display 30d, a power component 30e, an audio component 30f and other components. Figure 3 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 3 Components shown.

[0107] The detailed implementation and beneficial effects of the electronic device provided in the embodiments of the present application have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0108] The exemplary embodiments of the present application further provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is caused to implement the steps in the above-mentioned method embodiments.

[0109] An exemplary embodiment of the present application further provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the processor is enabled to implement the steps in the above-mentioned method embodiments.

[0110] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0111] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0112] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0113] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0114] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0115] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage, etc.) containing computer-usable program code.

[0116] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0117] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0119] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0120] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0121] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (Phase-change Random Access Memory, PRAM), static random access memory (SRAM), dynamic random access memory (Dynamic Random Access Memory, DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory, etc.

[0122] (EEPROM), flash memory or other memory technology, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0123] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0124] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A model conversion method, characterized in that: Applied to a service terminal, the method comprises: Receiving a model conversion request message sent by a mobile terminal, wherein the model conversion request message includes at least a system type of the mobile terminal, available resource information, and a model function required by the mobile terminal; Determining an initial digital human reasoning model adapted to the system type and required model functions of the mobile terminal from a digital human reasoning model developed based on a target model development framework, wherein the target model development framework refers to a model development framework applicable to the server; In the case where the available resource information of the mobile terminal meets the resource loading conditions required by the data packet, determining the target format conversion tool according to the original model format supported by the initial digital human reasoning model and the model-format conversion tool mapping table, wherein the model-format conversion tool mapping table stores format conversion tools corresponding to digital human reasoning models supporting different model formats; The target format conversion tool is called to convert the model format of the initial digital human reasoning model to obtain a target digital human reasoning model that supports the target model format, and the target digital human reasoning model is suitable for running on the mobile terminal.

2. The method according to claim 1, characterized in that Determining an initial digital human reasoning model adapted to the system type and required model functions of the mobile terminal from the digital human reasoning model developed based on the target model development framework includes: Selecting a first candidate digital human reasoning model adapted to the system type of the mobile terminal from the digital human reasoning models developed based on the target model development framework, wherein different digital human reasoning models are configured with a list of operating systems adapted thereto; Determine, based on the functional description information of the first candidate digital human reasoning model, a second candidate digital human reasoning model adapted to the model function required by the mobile terminal; If there is one second candidate digital human reasoning model, the second candidate digital human reasoning model is used as the initial digital human reasoning model; if there are multiple second candidate digital human reasoning models, the one with the smallest model file is selected as the initial digital human reasoning model.

3. The method according to claim 1, characterized in that Also includes: Obtaining a model file of the initial digital human reasoning model; According to the size of the model file, determining whether the bandwidth information and / or memory information in the available resource information of the mobile terminal meets the minimum bandwidth value and / or minimum memory amount required in the resource loading condition; If so, it is determined that the available resource information of the mobile terminal meets the resource loading condition.

4. The method according to claim 1, characterized in that Calling the target format conversion tool to convert the initial digital human reasoning model into a model format, so as to obtain a target digital human reasoning model supporting the target model format, including: Calling the target format conversion tool to parse the model file of the initial digital human reasoning model to obtain the network structure and weight parameter set of the initial digital human reasoning model, wherein the network structure is used to describe the multiple network layers included in the initial digital human reasoning model and the connection relationship between the multiple network layers, and the weight parameter set includes the original weight parameters of each network layer; And, by performing simulation operation or control flow analysis on the initial digital human reasoning model, a static calculation graph of the initial digital human reasoning model is obtained, wherein the static calculation graph includes the processing logic of each network layer from input data to output data and the data flow dependency relationship between network layers; According to the target model format, the network structure, weight parameter set and static calculation graph of the initial digital human reasoning model are reorganized to obtain the target digital human reasoning model.

5. The method according to claim 4, characterized in that The model file of the initial digital human reasoning model is parsed to obtain the network structure and weight parameter set of the initial digital human reasoning model, including: Parsing a subfile having a specific format in the model file to obtain a weight parameter set of the initial digital human reasoning model; The model structure of the initial digital human reasoning model is defined based on the forward propagation function in the model file to obtain the network structure of the initial digital human reasoning model.

6. The method according to claim 4, characterized in that The initial digital human reasoning model is simulated and run to obtain a static calculation graph of the initial digital human reasoning model, including: Inputting preset exemplary input data into the initial digital human reasoning model to simulate and run the initial digital human reasoning model; During the simulation operation, tracking and recording the computing operations involved in each network layer in the initial digital human reasoning model and the dependencies between the computing operations; The static computation graph is constructed according to the computation operations involved in each network layer and the dependencies between the computation operations.

7. The method according to claim 4, characterized in that By performing control flow analysis on the initial digital human reasoning model, a static calculation graph of the initial digital human reasoning model is obtained, including: By compiling the forward propagation function in the model file of the initial digital human reasoning model, the control flow statements and calculation operations in the forward propagation function are converted into an initial calculation graph; Analyze the dependencies between the computing operations, and add the dependencies to the initial computing graph to obtain the static computing graph.

8. The method according to any one of claims 4 to 7, characterized in that: According to the target model format, the network structure, weight parameter set and static calculation graph of the initial digital human reasoning model are reorganized to obtain the target digital human reasoning model, including: Based on the operator conversion table, the operators in each network layer in the network structure are converted into target operators in the target model format, wherein the operators are used to implement the calculation operations involved in the network layer; the operator conversion table stores the correspondence between the original operators in the original model format and the target operators in the target model format; Based on the weight parameter conversion relationship, the original weight parameters of each network layer in the weight parameter set are converted into the target weight parameters in the target model format; the weight parameter conversion relationship stores the corresponding relationship between the original weight parameters in the original model format and the target weight parameters in the target model format; Performing optimization operations on the network layers, the data flow dependencies between the network layers, and / or the target operators in the network layers in the network structure according to the static computation graph to obtain optimized network layers, the data flow dependencies between the network layers, and / or the target operations in the network layers as optimization objects; The target digital human reasoning model is generated according to the optimization object, the target weight parameter and the unoptimized object, wherein the unoptimized object refers to the network layer in the network structure, the data flow dependency between the network layers and the unoptimized objects in the target operator in the network layer.

9. The method according to claim 8, characterized in that Based on the operator conversion table, converting the operators in each network layer in the network structure into target operators in the target model format includes: Traversing the primitive operators in each network layer in the network structure; For the original operator currently traversed, query the operator conversion table; If the original operator currently traversed is found in the operator conversion table, the original operator currently traversed is replaced with the target operator corresponding thereto; If the currently traversed original operator is not found in the operator conversion table, the currently traversed original operator is replaced with a custom operator that supports the target model format.

10. The method according to any one of claims 1 to 9, characterized in that: The initial digital human reasoning model is a neural network model for generating a plurality of facial region images with continuously changing mouth shapes based on audio feature information; the method further comprises: Acquire sample audio data, a plurality of marked facial region images with continuously changing mouth shapes corresponding to the sample audio data, and a sample frame rate; Converting the sample audio data into sample audio feature information required by the basic digital human reasoning model, and segmenting the sample audio feature information according to the duration of the sample audio data and the sample frame rate to obtain a plurality of sample audio feature segments; Inputting the multiple sample audio feature segments into the basic digital human reasoning model, mapping the multiple sample audio feature segments into multiple sample mouth deformation features, and generating multiple sample face area images with continuously changing mouth shapes based on the multiple sample mouth deformation features; Calculate the loss function between the multiple sample facial area images and the multiple labeled facial area images, and if the loss function does not reach the set model training termination condition, adjust the model parameters of the basic digital human reasoning model until the loss function reaches the set model training termination condition to obtain the initial digital human reasoning model.

11. An electronic device, characterized in that: include: Memory and processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the method according to any one of claims 1 to 10.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to implement the steps in the method according to any one of claims 1 to 10.

13. A computer program product, characterized in that The computer program product comprises a computer program / instruction, which, when executed by a processor, enables the processor to implement the steps of any one of the methods of claims 1-10.

Images