Digital human video generation method and device based on artificial intelligence, equipment and medium

Abstract

The invention belongs to the field of artificial intelligence and financial science and technology, and relates to a digital human video generation method and device based on artificial intelligence, computer equipment and a storage medium, and the method comprises the steps: obtaining a face image and a verbal skill text inputted by a user; performing three-dimensional face reconstruction on the face image based on the hierarchical representation model to obtain a three-dimensional face model; extracting three-dimensional face model parameters from the three-dimensional face model; obtaining a target three-dimensional face model parameter based on adjustment processing of the three-dimensional face model parameter triggered by a user; performing voice generation processing on the verbal skill text based on a voice synthesis component to obtain target voice; performing digital human video synthesis on the target three-dimensional human face model parameters and the target voice based on a synthesis model to obtain a target digital human video; and returning the target digital human video to the user. In addition, the target digital human video may be stored in a blockchain. Based on the use of the hierarchical representation model and the synthesis model, the reality sense of the generated digital human video is effectively improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence development technology and financial technology, and in particular to methods, devices, computer equipment and storage media for generating digital human videos based on artificial intelligence. Background Art

[0002] With the rapid development of artificial intelligence technology, face reconstruction technology, as one of the core technologies in the field of image processing, has shown great application potential in the field of intelligent devices and digital human generation. However, although many financial companies are currently committed to the research and application of digital human generation technology, this field still faces many challenges, resulting in the lack of realism, naturalness and diversity of the generated digital humans.

[0003] At present, the three-dimensional face model (3DMM) is one of the most widely used digital human modeling methods. The model builds a basic three-dimensional face representation by learning a large amount of face image data, and generates a new face model through linear or nonlinear combination. However, the low-dimensional representation capability of 3DMM limits its ability to accurately express high-frequency information, such as subtle facial features such as wrinkles and dimples. This results in the generated digital faces being too smooth and lacking in realism, making it difficult to meet application scenarios that require high detail richness. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a method, device, computer equipment and storage medium for generating digital human videos based on artificial intelligence, so as to solve the technical problem that the digital human faces generated by the existing digital human modeling method based on three-dimensional face models are too smooth and lack of realism.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a method for generating digital human videos based on artificial intelligence, which adopts the following technical solutions:

[0006] Get the face image and speech text input by the user;

[0007] Performing three-dimensional face reconstruction on the face image based on a pre-constructed hierarchical representation model to obtain a corresponding three-dimensional face model; wherein the three-dimensional face model is constructed based on low-frequency geometric shape features, medium-frequency detail features, and high-frequency detail features corresponding to the face image;

[0008] Extracting three-dimensional face model parameters from the three-dimensional face model;

[0009] Based on the adjustment processing of the three-dimensional face model parameters triggered by the user, corresponding target three-dimensional face model parameters are obtained;

[0010] Performing speech generation processing on the speech text based on a preset speech synthesis component to obtain a target speech corresponding to the speech text;

[0011] Performing digital human video synthesis processing on the target three-dimensional face model parameters and the target voice based on a preset synthesis model to obtain a corresponding target digital human video;

[0012] The target digital human video is returned to the user.

[0013] Furthermore, the hierarchical representation model at least includes an input layer, a feature extraction layer, a decoupling module, a reconstruction layer, a fusion layer and an output layer; the step of performing three-dimensional face reconstruction on the face image based on the pre-constructed hierarchical representation model to obtain a corresponding three-dimensional face model specifically includes:

[0014] Inputting the face image into the input layer of the hierarchical representation model;

[0015] Extracting features from the face image through the feature extraction layer to obtain corresponding facial features;

[0016] Decoupling the facial features into initial low-frequency geometric shape features, initial mid-frequency detail features and initial high-frequency detail features through the decoupling module;

[0017] Reconstructing the initial low-frequency geometric shape features, the initial intermediate-frequency detail features, and the initial high-frequency detail features through the reconstruction layer to obtain corresponding low-frequency geometric shape features, intermediate-frequency detail features, and high-frequency detail features;

[0018] The low-frequency geometric shape features, the medium-frequency detail features and the high-frequency detail features are fused through the fusion layer to obtain a corresponding three-dimensional face model;

[0019] The three-dimensional face model is output through the output layer.

[0020] Furthermore, before the step of performing three-dimensional face reconstruction on the face image based on the pre-built hierarchical representation model to obtain a corresponding three-dimensional face model, the method further includes:

[0021] Obtain pre-built face image sample data;

[0022] Calling a preset initial hierarchical representation model;

[0023] Determining a target optimization algorithm corresponding to the initial hierarchical representation model;

[0024] Based on the target optimization algorithm and the preset contour perception loss function, the initial hierarchical representation model is trained using the facial image sample data to obtain a trained designated model;

[0025] The specified model is used as the hierarchical representation model.

[0026] Furthermore, the step of obtaining corresponding target three-dimensional face model parameters based on the adjustment processing of the three-dimensional face model parameters triggered by the user specifically includes:

[0027] Calling a preset parameter adjustment interface; wherein the parameter adjustment interface includes the three-dimensional face model parameters;

[0028] Displaying the parameter adjustment interface and receiving the parameter adjustment operation input by the user in the parameter adjustment interface;

[0029] Adjusting the three-dimensional face model parameters based on the parameter adjustment operation to obtain adjusted three-dimensional face model parameters;

[0030] The adjusted three-dimensional face model parameters are used as the target three-dimensional face model parameters.

[0031] Furthermore, the step of performing speech generation processing on the speech text based on the preset speech synthesis component to obtain a target speech corresponding to the speech text specifically includes:

[0032] Calling the speech synthesis component;

[0033] Performing text analysis on the speech text based on the speech synthesis component to obtain corresponding text analysis results;

[0034] Based on the text analysis result, calling a preset speech generation algorithm to perform speech generation processing on the speech text to obtain a corresponding first speech;

[0035] The target speech is generated based on the first speech.

[0036] Furthermore, the step of generating the target speech based on the first speech specifically includes:

[0037] Obtain preset parameter adjustment strategies and optimization strategies;

[0038] Performing parameter adjustment processing on the first speech based on the parameter adjustment strategy to obtain a corresponding second speech;

[0039] Optimize the second speech based on the optimization strategy to obtain a corresponding third speech;

[0040] The third speech is used as the target speech.

[0041] Furthermore, after the step of performing digital human video synthesis processing on the target 3D face model parameters and the target voice based on the preset synthesis model to obtain the corresponding target digital human video, the method further includes:

[0042] Determining a designated storage method corresponding to the target digital human video;

[0043] Calling a designated storage medium corresponding to the designated storage method;

[0044] The target digital human video is stored based on the designated storage medium.

[0045] In order to solve the above technical problems, the embodiment of the present application also provides a digital human video generation device based on artificial intelligence, which adopts the following technical solution:

[0046] The first acquisition module is used to acquire the face image and speech text input by the user;

[0047] A reconstruction module, used for performing three-dimensional face reconstruction on the face image based on a pre-constructed hierarchical representation model to obtain a corresponding three-dimensional face model; wherein the three-dimensional face model is constructed based on low-frequency geometric shape features, medium-frequency detail features and high-frequency detail features corresponding to the face image;

[0048] An extraction module, used to extract three-dimensional face model parameters from the three-dimensional face model;

[0049] An adjustment module, configured to obtain corresponding target three-dimensional face model parameters based on the adjustment processing of the three-dimensional face model parameters triggered by the user;

[0050] A generation module, used for performing speech generation processing on the speech text based on a preset speech synthesis component to obtain a target speech corresponding to the speech text;

[0051] A synthesis module, used for performing digital human video synthesis processing on the target three-dimensional face model parameters and the target voice based on a preset synthesis model to obtain a corresponding target digital human video;

[0052] The returning module is used to return the target digital human video to the user.

[0053] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0054] Get the face image and speech text input by the user;

[0055] Performing three-dimensional face reconstruction on the face image based on a pre-constructed hierarchical representation model to obtain a corresponding three-dimensional face model; wherein the three-dimensional face model is constructed based on low-frequency geometric shape features, medium-frequency detail features, and high-frequency detail features corresponding to the face image;

[0056] Extracting three-dimensional face model parameters from the three-dimensional face model;

[0057] Based on the adjustment processing of the three-dimensional face model parameters triggered by the user, corresponding target three-dimensional face model parameters are obtained;

[0058] Performing speech generation processing on the speech text based on a preset speech synthesis component to obtain a target speech corresponding to the speech text;

[0059] Performing digital human video synthesis processing on the target three-dimensional face model parameters and the target voice based on a preset synthesis model to obtain a corresponding target digital human video;

[0060] The target digital human video is returned to the user.

[0061] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0062] Get the face image and speech text input by the user;

[0063] Performing three-dimensional face reconstruction on the face image based on a pre-constructed hierarchical representation model to obtain a corresponding three-dimensional face model; wherein the three-dimensional face model is constructed based on low-frequency geometric shape features, medium-frequency detail features, and high-frequency detail features corresponding to the face image;

[0064] Extracting three-dimensional face model parameters from the three-dimensional face model;

[0065] Based on the adjustment processing of the three-dimensional face model parameters triggered by the user, corresponding target three-dimensional face model parameters are obtained;

[0066] Performing speech generation processing on the speech text based on a preset speech synthesis component to obtain a target speech corresponding to the speech text;

[0067] Performing digital human video synthesis processing on the target three-dimensional face model parameters and the target voice based on a preset synthesis model to obtain a corresponding target digital human video;

[0068] The target digital human video is returned to the user.

[0069] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0070] The present application first obtains a face image and a speech text input by a user; then, based on a pre-constructed hierarchical representation model, the face image is reconstructed in three dimensions to obtain a corresponding three-dimensional face model; wherein, the three-dimensional face model is constructed based on low-frequency geometric shape features, medium-frequency detail features, and high-frequency detail features corresponding to the face image; then, three-dimensional face model parameters are extracted from the three-dimensional face model; and based on the adjustment processing of the three-dimensional face model parameters triggered by the user, the corresponding target three-dimensional face model parameters are obtained; subsequently, based on a preset speech synthesis component, speech generation processing is performed on the speech text to obtain a target speech corresponding to the speech text; further, based on a preset synthesis model, a digital human video synthesis processing is performed on the target three-dimensional face model parameters and the target speech to obtain a corresponding target digital human video; finally, the target digital human video is returned to the user. The present application is based on the use of a hierarchical representation model and introduces three hierarchical representations, namely, low-frequency geometric shapes, medium-frequency details, and high-frequency details, which can more comprehensively capture the features of each level of the face. By constructing a corresponding 3D face model based on the low-frequency geometric shape features, mid-frequency detail features, and high-frequency detail features corresponding to the face image input by the user, the accuracy and detail restoration of the 3D face reconstruction are effectively improved, and the realism of the generated 3D face model is improved. In the subsequent process, by using the synthetic model to perform digital human video synthesis processing on the target 3D face model parameters corresponding to the 3D face model and the target speech of the speech text input by the user, a highly realistic target digital human video can be generated, which effectively improves the realism of the generated target digital human video. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0072] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0073] Figure 2 A flowchart of an embodiment of a method for generating digital human videos based on artificial intelligence according to the present application;

[0074] Figure 3 is a structural schematic diagram of an embodiment of a digital human video generation device based on artificial intelligence according to the present application;

[0075] Figure 4It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0077] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0078] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0079] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0080] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0081] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, and the like.

[0082] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0083] It should be noted that the artificial intelligence-based digital human video generation method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the artificial intelligence-based digital human video generation device is generally set in the server / terminal device.

[0084] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0085] Continue to refer Figure 2 , showing a flowchart of an embodiment of the method for generating digital human videos based on artificial intelligence according to the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. The method for generating digital human videos based on artificial intelligence provided in the embodiment of the present application can be applied to any scenario where digital human video generation is required, and the method for generating digital human videos based on artificial intelligence can be applied to products in these scenarios, for example, insurance digital human video generation in the financial insurance field. The method for generating digital human videos based on artificial intelligence comprises the following steps:

[0086] Step S201, obtaining the face image and speech text input by the user.

[0087] In this embodiment, the method for generating a digital human video based on artificial intelligence is executed on an electronic device (e.g. Figure 1The server / terminal device shown in the figure) can obtain the face image and speech text input by the user through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, Wi-Fi connection, Bluetooth connection, Wi MAX connection, Zigbee connection, UWB (Ultra Wi DeBand) connection, and other wireless connection methods currently known or to be developed in the future. The execution subject of the present application may specifically be a digital human generation system, which may also be referred to as a system. The digital human video generation method based on artificial intelligence of the present application can be applied to the business scenario of insurance agency digital human generation in the field of finance and insurance. The user can input a face image according to the actual business needs of insurance agency digital human generation, and correspondingly input a speech text. The speech text may be an insurance speech text commonly used in insurance business, such as an opening remark, product introduction, customer question answering, etc.

[0088] Step S202: reconstructing a three-dimensional face on the face image based on a pre-built hierarchical representation model to obtain a corresponding three-dimensional face model.

[0089] In this embodiment, the above-mentioned three-dimensional face model is constructed based on the low-frequency geometric shape features, medium-frequency detail features and high-frequency detail features corresponding to the face image. By using the hierarchical representation model, the three-dimensional face can be accurately and detailedly reconstructed from the face image provided by the user. By decoupling the head geometry into three key parts and introducing three hierarchical representations, a multi-level and highly accurate reconstruction of the head is achieved. The method based on hierarchical representation comprehensively considers the overall skeleton of the head, large-scale details of the face and texture details at the pixel level, and the decoupling method makes modeling at different levels more flexible, can better adapt to different scales and details customization, improves the freedom and realism of reconstruction, and provides a more flexible basis for subsequent customization and adjustment. Among them, the above-mentioned three-dimensional face reconstruction of the face image based on the pre-constructed hierarchical representation model to obtain the corresponding three-dimensional face model is further described in detail in the subsequent specific embodiments of this application, and will not be elaborated on here.

[0090] Step S203: extracting 3D face model parameters from the 3D face model.

[0091] In this embodiment, after obtaining the three-dimensional face model output by the hierarchical representation model, appearance parameters, namely the above-mentioned three-dimensional face model parameters, can be extracted from the three-dimensional face model. These parameters may include parameters such as facial shape, position of facial features, and texture information.

[0092] Step S204: obtaining corresponding target three-dimensional face model parameters based on the adjustment processing of the three-dimensional face model parameters triggered by the user.

[0093] In this embodiment, the system also provides a customization function for appearance parameters, i.e., three-dimensional face model parameters. By adjusting the appearance parameters, it can be ensured that the generated digital person meets the user's needs in appearance. This can provide users with a highly personalized experience, allowing users to customize the appearance characteristics of the agent. Among them, the specific implementation process of the above adjustment process of the three-dimensional face model parameters triggered by the user to obtain the corresponding target three-dimensional face model parameters will be further described in detail in the subsequent specific embodiments of this application, and will not be elaborated on here.

[0094] Step S205, performing speech generation processing on the speech text based on a preset speech synthesis component to obtain a target speech corresponding to the speech text.

[0095] In this embodiment, the above-mentioned specific implementation process of performing speech generation processing on the speech text based on the preset speech synthesis component to obtain the target speech corresponding to the speech text will be described in further details in subsequent specific embodiments of the present application and will not be elaborated in detail here.

[0096] Step S206, performing digital human video synthesis processing on the target three-dimensional face model parameters and the target voice based on a preset synthesis model to obtain a corresponding target digital human video.

[0097] In this embodiment, the synthesis model may specifically adopt a general audio / face synthesis network. By inputting the target 3D face model parameters and the target voice into the synthesis model, the synthesis model will consider both audio and face information to achieve a more realistic and lifelike digital human performance, and perform audio-driven digital human video synthesis processing to output the corresponding target digital human video.

[0098] Step S207: returning the target digital human video to the user.

[0099] In this embodiment, the target digital human video can be returned to the user based on the communication information obtained from the user, wherein the communication information may include a mobile phone number or an email address.

[0100] The present application first obtains a face image and a speech text input by a user; then, based on a pre-constructed hierarchical representation model, the face image is reconstructed in three dimensions to obtain a corresponding three-dimensional face model; wherein, the three-dimensional face model is constructed based on low-frequency geometric shape features, medium-frequency detail features, and high-frequency detail features corresponding to the face image; then, three-dimensional face model parameters are extracted from the three-dimensional face model; and based on the adjustment processing of the three-dimensional face model parameters triggered by the user, the corresponding target three-dimensional face model parameters are obtained; subsequently, based on a preset speech synthesis component, speech generation processing is performed on the speech text to obtain a target speech corresponding to the speech text; further, based on a preset synthesis model, a digital human video synthesis processing is performed on the target three-dimensional face model parameters and the target speech to obtain a corresponding target digital human video; finally, the target digital human video is returned to the user. The present application is based on the use of a hierarchical representation model and introduces three hierarchical representations, namely, low-frequency geometric shapes, medium-frequency details, and high-frequency details, which can more comprehensively capture the features of each level of the face. By constructing a corresponding 3D face model based on the low-frequency geometric shape features, mid-frequency detail features, and high-frequency detail features corresponding to the face image input by the user, the accuracy and detail restoration of the 3D face reconstruction are effectively improved, and the realism of the generated 3D face model is improved. In the subsequent process, by using the synthetic model to perform digital human video synthesis processing on the target 3D face model parameters corresponding to the 3D face model and the target speech of the speech text input by the user, a highly realistic target digital human video can be generated, which effectively improves the realism of the generated target digital human video.

[0101] In some optional implementations, step S202 includes the following steps:

[0102] The face image is input into the input layer of the hierarchical representation model.

[0103] In this embodiment, the hierarchical representation model for reconstructing accurate faces includes multiple layers, each layer is responsible for capturing facial features of different scales, and the network structure of the hierarchical representation model includes at least an input layer, a feature extraction layer, a decoupling module, a reconstruction layer, a fusion layer, and an output layer.

[0104] The feature extraction layer extracts features from the face image to obtain corresponding facial features.

[0105] In this embodiment, the feature extraction layer includes multiple convolution blocks, each of which is composed of a convolution layer, an activation layer (such as ReLU) and a pooling layer. Facial features in an image can be extracted through a structure such as a convolutional neural network (CNN).

[0106] The facial features are decoupled into initial low-frequency geometric shape features, initial mid-frequency detail features and initial high-frequency detail features through the decoupling module.

[0107] In this embodiment, the extracted facial features can be decoupled into three parts, namely, low-frequency geometry, medium-frequency details and high-frequency details, by designing a specific network layer or module, so as to obtain the corresponding initial low-frequency geometry features, initial medium-frequency detail features and initial high-frequency detail features. Among them, the residual network (ResNet) structure can be used to capture features of different scales. Specifically, by introducing three hierarchical representations, namely, low-frequency geometry, medium-frequency details and high-frequency details, the features of each level of the face are captured more comprehensively, and the accuracy of reconstruction and the degree of detail restoration are improved. Low-frequency geometry: captures the overall skeleton and general outline of the head, usually corresponding to a coarser three-dimensional mesh. Medium-frequency details: describes the main shape features of the face, such as the position and shape of the eyes, nose and mouth, which are manifested as finer local structures on the three-dimensional mesh. High-frequency details: captures the subtle texture and expression changes of the face, such as wrinkles, dimples, etc. These features may be manifested in the form of vertex displacement maps or normal maps on the three-dimensional mesh.

[0108] The initial low-frequency geometric shape features, the initial intermediate-frequency detail features and the initial high-frequency detail features are reconstructed through the reconstruction layer to obtain corresponding low-frequency geometric shape features, intermediate-frequency detail features and high-frequency detail features.

[0109] In this embodiment, the reconstruction layer may include a deconvolution layer (also called a transposed convolution layer) or an upsampling layer to convert the feature map into a three-dimensional mesh or point cloud form. The reconstruction layer reconstructs the three-dimensional representations of low-frequency geometry, medium-frequency details and high-frequency details respectively according to the decoupled features to obtain corresponding low-frequency geometry features, medium-frequency detail features and high-frequency detail features. Alternatively, the reconstruction layer includes three layers: low-frequency reconstruction layer: uses shallower network layers or features after global average pooling to reconstruct low-frequency geometry, which usually corresponds to the overall skeleton and outline of the head. Medium-frequency reconstruction layer: uses medium-depth network layers to reconstruct medium-frequency details, such as the position and shape of eyes, nose and mouth. High-frequency reconstruction layer: uses deeper network layers or specific high-frequency capture modules to reconstruct high-frequency details, such as wrinkles, dimples and other tiny textures. In addition, the representation of the reconstruction result can be a three-dimensional mesh, point cloud or volume representation. For mesh representation, a graph convolutional network (GCN) or a parameterized mesh deformation method can be used; for point cloud representation, structures such as PointNet can be used; for volume representation, a three-dimensional convolutional network can be used.

[0110] The low-frequency geometric shape features, the medium-frequency detail features and the high-frequency detail features are fused through the fusion layer to obtain a corresponding three-dimensional face model;

[0111] In this embodiment, the low-frequency geometric shape features, the medium-frequency detail features and the high-frequency detail features are fused into a complete three-dimensional face model through the fusion layer. The fusion layer can be implemented by weighted summation or feature-level fusion strategy. Weighted summation: Add the reconstruction results of different scales according to predefined weights or learned weights. Feature-level fusion strategy: Fusion is performed at the feature level, such as using concatenation or element-wise multiplication.

[0112] The three-dimensional face model is output through the output layer.

[0113] In this embodiment, the output form of the three-dimensional face model may be a grid, a point cloud, or a volume representation.

[0114] The present application inputs the face image into the input layer of the hierarchical representation model; then extracts features from the face image through the feature extraction layer to obtain corresponding facial features; then decouples the facial features into initial low-frequency geometric shape features, initial mid-frequency detail features and initial high-frequency detail features through the decoupling module; subsequently reconstructs the initial low-frequency geometric shape features, the initial mid-frequency detail features and the initial high-frequency detail features through the reconstruction layer to obtain corresponding low-frequency geometric shape features, mid-frequency detail features and high-frequency detail features; further fuses the low-frequency geometric shape features, the mid-frequency detail features and the high-frequency detail features through the fusion layer to obtain a corresponding three-dimensional face model; finally, outputs the three-dimensional face model through the output layer. Based on the use of the hierarchical representation model, the present application introduces three hierarchical representations, namely low-frequency geometric shapes, mid-frequency details and high-frequency details, to more comprehensively capture the features of each level of the face, effectively improve the accuracy and detail restoration of the three-dimensional face reconstruction, and improve the realism of the generated three-dimensional face model.

[0115] In some optional implementations of this embodiment, before step S202, the electronic device may further perform the following steps:

[0116] Get pre-built face image sample data.

[0117] In this embodiment, the above-mentioned facial image sample data is obtained by collecting and annotating a three-dimensional facial data set, which includes two-dimensional facial images under different angles, expressions and lighting conditions and their corresponding three-dimensional facial models.

[0118] Call the preset initial hierarchical representation model.

[0119] In this embodiment, the initial hierarchical representation model includes multiple levels, each level is responsible for capturing facial features of different scales, and the network structure of the initial hierarchical representation model includes at least an input layer, a feature extraction layer, a decoupling module, a reconstruction layer, a fusion layer and an output layer.

[0120] A target optimization algorithm corresponding to the initial hierarchical representation model is determined.

[0121] In this embodiment, there is no specific limitation on the selection of the above-mentioned target optimization algorithm, which can be determined according to the actual model building requirements. For example, the back propagation algorithm can be selected.

[0122] Based on the target optimization algorithm and the preset contour perception loss function, the initial hierarchical representation model is trained using the facial image sample data to obtain a trained designated model.

[0123] In this embodiment, the initial hierarchical representation model is trained by using the above-mentioned face image sample data, and the model parameters are optimized by a target optimization algorithm to minimize the contour perception loss function, thereby obtaining a trained designated model.

[0124] Specifically, in the training process of the hierarchical representation model, contour-aware loss is used to achieve accurate modeling of facial contours. Contour-aware loss acts on the contour edge training of face reconstruction, aiming to pull the vertices of its edges to align the facial contour. First, the contour vertices of the reconstructed face are projected into the image space. Then the pre-trained face matting network is used to predict the facial mask and post-process to obtain the left and right points of each row of the face. Given the vertex p and the corresponding projection point p' on the predicted face mask, the vector l is obtained. p and r p (From p' to the edge point in the horizontal direction). Contour perception loss (L con ) can be described as:

[0125]

[0126] Among them, N p: Represents the total number of contour vertices projected into the image space. Used to calculate the average value in order to obtain the average value of the loss on all contour vertices. M1: Represents the set of contour vertices projected into the image space. The summation operation in the formula is performed on this set. p: Represents a vertex on the contour (in three-dimensional space). During the training process, this vertex will be projected into the image space to obtain the corresponding projection point p′. p′: Represents the projection point of vertex p in the image space. Used to interact with the predicted facial mask to determine whether the vertex is located on the facial contour. y(p′): Represents the value of the projected point p′ on the predicted facial mask. This value is used to determine whether p′ is within the facial area (ie, y(p′)>δ). δ: It is a threshold used to determine whether the projection point is within the facial area. Only when y(p′)>δ, the corresponding vertex p will be counted in the calculation of contour-aware loss. is an indicator function, which is 1 when the condition in the brackets is met, otherwise it is 0. Here, it is used to determine whether the projection point is within the facial region, and decide whether to calculate the loss of the vertex accordingly. p and r p : represents the vectors from the projection point p′ to the left and right edge points of the corresponding row on the face mask. These vectors are used to calculate the relative position relationship between the vertex p and the facial contour. h(·): is a function used to calculate the vector l p and r p Some relationship or measure between (maybe angle, distance, etc.). The specific form may vary depending on implementation details. λ: is a threshold or adjustment parameter used to control the tightness of the contour or the sensitivity of the loss function. By adjusting the value of λ, the vertices on the facial contour can be pulled to align with the true contour while keeping the internal vertices of the face fixed.

[0127] The contour-aware loss penalizes vertices outside the soft boundary of the face and pulls them to the face contour, while keeping the vertices inside the face unchanged. This loss function only focuses on the lower part of the facial contour to avoid the interference of hair, which can effectively accelerate the convergence of the model. The parameters in the contour-aware loss formula jointly define how to calculate and optimize the position of vertices on the facial contour to achieve more accurate face reconstruction. The contour-aware loss formula is used to calculate the difference between the position of vertices on the facial contour and the true contour, and improve the accuracy of face reconstruction by optimizing this loss.

[0128] The specified model is used as the hierarchical representation model.

[0129] This application uses a determined target optimization algorithm and a preset contour perception loss function, and uses pre-built facial image sample data to train the initial hierarchical representation model, so as to realize the rapid and intelligent construction of the required hierarchical representation model, thereby improving the construction efficiency of the hierarchical representation model. In addition, the use of contour perception loss can realize accurate modeling of facial contours, effectively accelerate the model convergence of the hierarchical representation model, ensure the model effect of the generated hierarchical representation model, and improve the accuracy of the face reconstruction processing of the hierarchical representation model.

[0130] In some optional implementations, step S204 includes the following steps:

[0131] Calling a preset parameter adjustment interface; wherein the parameter adjustment interface includes the three-dimensional face model parameters.

[0132] In this embodiment, a user-friendly parameter adjustment interface is pre-constructed to allow the user to intuitively adjust the appearance parameters of the 3D face model, that is, the 3D face model parameters.

[0133] The parameter adjustment interface is displayed, and a parameter adjustment operation input by the user in the parameter adjustment interface is received.

[0134] In this embodiment, the parameter adjustment interface can provide a variety of interactive modes such as sliders and selectors to facilitate users to customize the appearance parameters of the 3D face model. The parameter adjustment operations include adding, deleting, replacing, resetting, and other operations.

[0135] The three-dimensional face model parameters are adjusted based on the parameter adjustment operation to obtain adjusted three-dimensional face model parameters.

[0136] In this embodiment, the parameters of the 3D face model can be adjusted to match the parameter adjustment operation input by the user to ensure that the adjusted 3D face model parameters meet the user's needs. When the user adjusts the 3D face model parameters, a preview of the adjusted 3D face model is generated and displayed in real time so that the user can intuitively see the adjustment effect.

[0137] The adjusted three-dimensional face model parameters are used as the target three-dimensional face model parameters.

[0138] In this embodiment, after the user completes the customization of the appearance parameters of the 3D face model, the adjusted appearance parameters can be saved in the system. When subsequently generating a digital human video, these customized appearance parameters can be applied to ensure that the generated digital human meets the user's requirements in appearance.

[0139] Since human faces require copyright, this system can adjust the parameters of the three-dimensional face model to create countless new faces and achieve personalized customization of appearance parameters without the need for additional customized copyrights, thereby increasing the diversity of generated digital humans and reducing the cost of building digital humans.

[0140] This application provides users with a highly personalized appearance customization service for a three-dimensional face model based on the use of a parameter adjustment interface. By adjusting the parameters of the three-dimensional face model based on the user's triggering in the parameter adjustment interface, it is possible to quickly and accurately obtain the target three-dimensional face model parameters that meet the user's needs, effectively enhancing the user experience and meeting the diverse needs of users. And when subsequently generating a digital human video, by applying the customized target three-dimensional face model parameters, it can be ensured that the generated digital human meets the user's needs in appearance.

[0141] In some optional implementations, step S205 includes the following steps:

[0142] The speech synthesis component is called.

[0143] In this embodiment, the speech synthesis component may adopt a speech synthesis module that has end-to-end processing capabilities and can ensure that the generated speech is fluent and natural.

[0144] Based on the speech synthesis component, text analysis is performed on the speech text to obtain corresponding text analysis results.

[0145] In this embodiment, the above-mentioned text analysis includes word segmentation, part-of-speech tagging, syntactic analysis and other processing. By performing text analysis on the speech text through the speech synthesis component, the meaning and structure of the speech text can be better understood, and corresponding text analysis results can be generated.

[0146] Based on the text analysis result, a preset speech generation algorithm is called to perform speech generation processing on the speech text to obtain a corresponding first speech.

[0147] In this embodiment, there is no specific limitation on the selection of the above-mentioned speech generation algorithm, which can be set according to actual usage requirements. For example, synthesis based on statistical parameters, synthesis based on unit selection and waveform splicing, or end-to-end deep learning methods can be adopted.

[0148] The target speech is generated based on the first speech.

[0149] In this embodiment, the specific implementation process of generating the target speech based on the first speech will be further described in detail in subsequent specific embodiments of the present application, and will not be elaborated on here.

[0150] The present application performs text analysis on the speech text through a speech synthesis component to obtain a corresponding text analysis result, and then based on the text analysis result, calls a preset speech generation algorithm to perform speech generation processing on the speech text to obtain a first speech, and then generates a final target speech based on the obtained first speech, so as to realize end-to-end speech synthesis processing of the speech text, and generate a smooth and natural speech output that matches the image of a digital human, thereby improving the accuracy and generation intelligence of the target speech.

[0151] In some optional implementations of this embodiment, generating the target speech based on the first speech includes the following steps:

[0152] Get preset parameter adjustment strategies and optimization strategies.

[0153] In this embodiment, the policy content of the parameter adjustment policy may include adjusting the speech generation parameters of the dialogue text according to actual business needs, such as timbre, speech speed, intonation, etc., to ensure that the generated speech meets the needs of specific scenarios and audiences. The policy content of the optimization policy may include performing sound quality optimization processing on the generated speech, such as removing noise, adjusting volume balance, etc., to further improve the clarity and audibility of the speech.

[0154] The first speech is subjected to parameter adjustment processing based on the parameter adjustment strategy to obtain a corresponding second speech.

[0155] In this embodiment, the first voice may be subjected to matching parameter adjustment processing according to the policy content of the above parameter adjustment policy, thereby obtaining a corresponding second voice.

[0156] The second speech is optimized based on the optimization strategy to obtain a corresponding third speech.

[0157] In this embodiment, the second voice may be optimized according to the policy content of the above optimization policy, so as to obtain the corresponding third voice.

[0158] The third speech is used as the target speech.

[0159] After calling the preset speech generation algorithm to perform speech generation processing on the speech text to obtain the corresponding first speech, the present application will also intelligently perform parameter adjustment and optimization processing on the first speech based on the use of parameter adjustment strategy and optimization strategy, thereby realizing end-to-end speech synthesis processing of the speech text, and generating smooth and natural speech output that matches the image of the digital human, thereby improving the accuracy and generation intelligence of the target speech.

[0160] In some optional implementations of this embodiment, after step S206, the electronic device may further perform the following steps:

[0161] Determine a designated storage mode corresponding to the target digital human video.

[0162] In this embodiment, the selection of the above-mentioned designated storage method is not specifically limited and can be determined according to actual business usage requirements. Exemplarily, the designated storage method can adopt any one of local database storage, network disk storage, cloud server storage, blockchain storage, etc. Blockchain storage is preferably used as the above-mentioned designated storage method. By using the blockchain storage method to store and manage the target digital human video, the security and non-tamperability of the target digital human video can be effectively guaranteed.

[0163] The designated storage medium corresponding to the designated storage mode is called.

[0164] In this embodiment, for storage methods such as local database storage, network disk storage, cloud server storage, blockchain storage, etc., the storage medium corresponding to the storage method may include a local database, a network disk, a cloud server, and a blockchain.

[0165] The target digital human video is stored based on the designated storage medium.

[0166] In this embodiment, the storage location of the designated storage medium can be determined, and then a communication connection relationship can be established with the designated storage medium according to the storage location, so that the target digital human video can be stored in the designated storage medium.

[0167] After the target 3D face model parameters and the target voice are synthesized into a digital human video based on a preset synthesis model to obtain the corresponding target digital human video, the application will also intelligently determine the designated storage method corresponding to the target digital human video, and call the designated storage medium corresponding to the designated storage method to store the target digital human video, so as to effectively ensure the data security and non-tamperability of the target digital human video, thereby improving the storage intelligence of the target digital human video. In addition, by storing the target digital human video, the required digital human video can be directly called from the designated storage medium in the future without the need to regenerate the digital human video, thereby improving the user's work convenience.

[0168] In the business scenario of digital human generation for insurance agents in the financial insurance field, this application can reconstruct a single face image, edit feature parameters, and realize highly realistic digital human generation for insurance agents, thereby improving the user interaction experience, improving the overall work efficiency of insurance Internet sales, reducing labor costs, and ensuring the service quality of insurance sales. In addition, this application improves the naturalness of audio and face synthesis and the coherence of head and expression movement of digital humans of insurance agents by introducing more feature dimensions. In addition, this system reduces the cost of image modeling. By changing the facial feature parameters, it is possible to generate digital humans of different images with only one face image. It is no longer necessary to use real-life photos or videos taken or recorded in advance for digital human generation, reducing copyright costs. This application not only improves the authenticity of digital humans of insurance agents, but also opens up new possibilities for technology and applications in the field of digital humans.

[0169] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0170] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned target digital human video, the above-mentioned target digital human video can also be stored in a node of a blockchain.

[0171] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0172] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0173] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0174] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0175] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0176] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a digital human video generation device based on artificial intelligence, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0177] like Figure 3 As shown, the artificial intelligence-based digital human video generation device 300 described in this embodiment includes: a first acquisition module 301, a reconstruction module 302, an extraction module 303, an adjustment module 304, a generation module 305, a synthesis module 306 and a return module 307. Among them:

[0178] The first acquisition module 301 is used to acquire the face image and speech text input by the user;

[0179] A reconstruction module 302 is used to perform three-dimensional face reconstruction on the face image based on a pre-constructed hierarchical representation model to obtain a corresponding three-dimensional face model; wherein the three-dimensional face model is constructed based on low-frequency geometric shape features, medium-frequency detail features, and high-frequency detail features corresponding to the face image;

[0180] An extraction module 303, used to extract 3D face model parameters from the 3D face model;

[0181] An adjustment module 304, configured to obtain corresponding target 3D face model parameters based on the adjustment processing of the 3D face model parameters triggered by the user;

[0182] A generation module 305 is used to perform speech generation processing on the speech text based on a preset speech synthesis component to obtain a target speech corresponding to the speech text;

[0183] A synthesis module 306 is used to perform digital human video synthesis processing on the target 3D human face model parameters and the target voice based on a preset synthesis model to obtain a corresponding target digital human video;

[0184] The return module 307 is used to return the target digital human video to the user.

[0185] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the method for generating digital human videos based on artificial intelligence in the aforementioned implementation mode, and are not described in detail here.

[0186] In some optional implementations of this embodiment, the hierarchical representation model includes at least an input layer, a feature extraction layer, a decoupling module, a reconstruction layer, a fusion layer, and an output layer; the reconstruction module 302 includes:

[0187] An input submodule, used for inputting the face image into the input layer of the hierarchical representation model;

[0188] An extraction submodule, used for performing feature extraction on the face image through the feature extraction layer to obtain corresponding facial features;

[0189] a decoupling submodule, configured to decouple the facial features into initial low-frequency geometric shape features, initial mid-frequency detail features, and initial high-frequency detail features through the decoupling module;

[0190] A reconstruction submodule, configured to reconstruct the initial low-frequency geometric shape features, the initial intermediate-frequency detail features, and the initial high-frequency detail features through the reconstruction layer to obtain corresponding low-frequency geometric shape features, intermediate-frequency detail features, and high-frequency detail features;

[0191] A fusion submodule, configured to fuse the low-frequency geometric shape features, the medium-frequency detail features and the high-frequency detail features through the fusion layer to obtain a corresponding three-dimensional face model;

[0192] The output submodule is used to output the three-dimensional face model through the output layer.

[0193] In some optional implementations of this embodiment, the device for generating digital human video based on artificial intelligence further includes:

[0194] The second acquisition module is used to acquire pre-built face image sample data;

[0195] A first calling module, used for calling a preset initial hierarchical representation model;

[0196] A first determination module, used to determine a target optimization algorithm corresponding to the initial hierarchical representation model;

[0197] A training module, used to train the initial hierarchical representation model using the face image sample data based on the target optimization algorithm and a preset contour perception loss function to obtain a trained designated model;

[0198] The second determining module is used to use the specified model as the hierarchical representation model.

[0199] In some optional implementations of this embodiment, the adjustment module 304 includes:

[0200] A first calling submodule, used to call a preset parameter adjustment interface; wherein the parameter adjustment interface includes the three-dimensional face model parameters;

[0201] A receiving submodule, used to display the parameter adjustment interface and receive the parameter adjustment operation input by the user in the parameter adjustment interface;

[0202] An adjustment submodule, configured to adjust the three-dimensional face model parameters based on the parameter adjustment operation to obtain adjusted three-dimensional face model parameters;

[0203] The determination submodule is used to use the adjusted three-dimensional face model parameters as the target three-dimensional face model parameters.

[0204] In some optional implementations of this embodiment, the generating module 305 includes:

[0205] A second calling submodule, used for calling the speech synthesis component;

[0206] An analysis submodule, used for performing text analysis on the speech text based on the speech synthesis component to obtain corresponding text analysis results;

[0207] A processing submodule, for invoking a preset speech generation algorithm to perform speech generation processing on the speech text based on the text analysis result to obtain a corresponding first speech;

[0208] A generating submodule is used to generate the target speech based on the first speech.

[0209] In some optional implementations of this embodiment, the generating submodule includes:

[0210] An acquisition unit, used to acquire preset parameter adjustment strategies and optimization strategies;

[0211] An adjustment unit, configured to perform parameter adjustment processing on the first speech based on the parameter adjustment strategy to obtain a corresponding second speech;

[0212] An optimization unit, configured to optimize the second speech based on the optimization strategy to obtain a corresponding third speech;

[0213] A determination unit is used to use the third speech as the target speech.

[0214] In some optional implementations of this embodiment, the device for generating a digital human video based on artificial intelligence further includes:

[0215] A third determination module is used to determine a designated storage method corresponding to the target digital human video;

[0216] A second calling module, used to call a designated storage medium corresponding to the designated storage method;

[0217] The storage module is used to store the target digital human video based on the designated storage medium.

[0218] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0219] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0220] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0221] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a digital human video generation method based on artificial intelligence, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0222] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as running the computer-readable instructions of the artificial intelligence-based digital human video generation method.

[0223] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0224] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned artificial intelligence-based digital human video generation method.

[0225] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0226] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for generating digital human videos based on artificial intelligence, characterized in that: The steps include: Get the face image and speech text input by the user; Performing three-dimensional face reconstruction on the face image based on a pre-constructed hierarchical representation model to obtain a corresponding three-dimensional face model; wherein the three-dimensional face model is constructed based on low-frequency geometric shape features, medium-frequency detail features, and high-frequency detail features corresponding to the face image; Extracting three-dimensional face model parameters from the three-dimensional face model; Based on the adjustment processing of the three-dimensional face model parameters triggered by the user, corresponding target three-dimensional face model parameters are obtained; Performing speech generation processing on the speech text based on a preset speech synthesis component to obtain a target speech corresponding to the speech text; Performing digital human video synthesis processing on the target three-dimensional face model parameters and the target voice based on a preset synthesis model to obtain a corresponding target digital human video; The target digital human video is returned to the user.

2. The method for generating digital human video based on artificial intelligence according to claim 1, characterized in that: The hierarchical representation model at least includes an input layer, a feature extraction layer, a decoupling module, a reconstruction layer, a fusion layer and an output layer; the step of performing three-dimensional face reconstruction on the face image based on the pre-built hierarchical representation model to obtain a corresponding three-dimensional face model specifically includes: Inputting the face image into the input layer of the hierarchical representation model; Extracting features from the face image through the feature extraction layer to obtain corresponding facial features; Decoupling the facial features into initial low-frequency geometric shape features, initial mid-frequency detail features and initial high-frequency detail features through the decoupling module; Reconstructing the initial low-frequency geometric shape features, the initial intermediate-frequency detail features, and the initial high-frequency detail features through the reconstruction layer to obtain corresponding low-frequency geometric shape features, intermediate-frequency detail features, and high-frequency detail features; The low-frequency geometric shape features, the medium-frequency detail features and the high-frequency detail features are fused through the fusion layer to obtain a corresponding three-dimensional face model; The three-dimensional face model is output through the output layer.

3. The method for generating digital human video based on artificial intelligence according to claim 1, characterized in that: Before the step of performing three-dimensional face reconstruction on the face image based on the pre-built hierarchical representation model to obtain a corresponding three-dimensional face model, the method further includes: Obtain pre-built face image sample data; Calling a preset initial hierarchical representation model; Determining a target optimization algorithm corresponding to the initial hierarchical representation model; Based on the target optimization algorithm and the preset contour perception loss function, the initial hierarchical representation model is trained using the facial image sample data to obtain a trained designated model; The specified model is used as the hierarchical representation model.

4. The method for generating digital human video based on artificial intelligence according to claim 1, characterized in that: The step of obtaining corresponding target three-dimensional face model parameters based on the adjustment processing of the three-dimensional face model parameters triggered by the user specifically includes: Calling a preset parameter adjustment interface; wherein the parameter adjustment interface includes the three-dimensional face model parameters; Displaying the parameter adjustment interface and receiving the parameter adjustment operation input by the user in the parameter adjustment interface; Adjusting the three-dimensional face model parameters based on the parameter adjustment operation to obtain adjusted three-dimensional face model parameters; The adjusted three-dimensional face model parameters are used as the target three-dimensional face model parameters.

5. The method for generating digital human video based on artificial intelligence according to claim 1, characterized in that: The step of performing speech generation processing on the speech text based on a preset speech synthesis component to obtain a target speech corresponding to the speech text specifically includes: Calling the speech synthesis component; Performing text analysis on the speech text based on the speech synthesis component to obtain corresponding text analysis results; Based on the text analysis result, calling a preset speech generation algorithm to perform speech generation processing on the speech text to obtain a corresponding first speech; The target speech is generated based on the first speech.

6. The method for generating digital human video based on artificial intelligence according to claim 5, characterized in that: The step of generating the target speech based on the first speech specifically includes: Obtain preset parameter adjustment strategies and optimization strategies; Performing parameter adjustment processing on the first speech based on the parameter adjustment strategy to obtain a corresponding second speech; Optimizing the second speech based on the optimization strategy to obtain a corresponding third speech; The third speech is used as the target speech.

7. The method for generating digital human video based on artificial intelligence according to claim 1, characterized in that: After the step of performing digital human video synthesis processing on the target 3D face model parameters and the target voice based on the preset synthesis model to obtain the corresponding target digital human video, the method further includes: Determining a designated storage method corresponding to the target digital human video; Calling a designated storage medium corresponding to the designated storage method; The target digital human video is stored based on the designated storage medium.

8. A digital human video generation device based on artificial intelligence, characterized in that: include: The first acquisition module is used to acquire the face image and speech text input by the user; A reconstruction module, used for performing three-dimensional face reconstruction on the face image based on a pre-constructed hierarchical representation model to obtain a corresponding three-dimensional face model; wherein the three-dimensional face model is constructed based on low-frequency geometric shape features, medium-frequency detail features and high-frequency detail features corresponding to the face image; An extraction module, used to extract three-dimensional face model parameters from the three-dimensional face model; An adjustment module, configured to obtain corresponding target three-dimensional face model parameters based on the adjustment processing of the three-dimensional face model parameters triggered by the user; A generation module, used for performing speech generation processing on the speech text based on a preset speech synthesis component to obtain a target speech corresponding to the speech text; A synthesis module, used for performing digital human video synthesis processing on the target three-dimensional face model parameters and the target voice based on a preset synthesis model to obtain a corresponding target digital human video; The returning module is used to return the target digital human video to the user.

9. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the method for generating digital human videos based on artificial intelligence as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the method for generating digital human videos based on artificial intelligence as claimed in any one of claims 1 to 7.

Images